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ABSTRACT 


This  present  study  was  designed  to  develop  an  evaluation  of 
clinical  competence  with  physical  therapy  (PT)  students  and  to 
establish  its  reliability  and  validity.  The  study  of  clinical 
competence  in  PT  has  focused  primarily  on  defining  the  knowledge, 
skills,  judgements  and  attitudes  that  must  be  acquired  by  PT  students 
for  satisfactory  performance  upon  entering  professional  practice. 
Concomitant  with  the  emphasis  on  defining  clinical  competence  is  the 
need  for  salient  evaluation  and  diagnosis  of  clinical  competence  in  PT 
education. 

A  cri terion-ref erence  evaluation  instrument.  Evaluation  of 
Clinical  Competence  (ECC),  was  developed  by  following  the  procedures 
as  outlined.  (1)  Clinical  competencies  required  for  PT  practice  were 
identified  by  a  survey  of  the  PT  literature  and  professional  documents 
and,  then,  were  validated  by  a  questionnaire  completed  by  121  physical 
therapists  working  in  Edmonton.  The  questionnaire  results  also 
provided  a  basis  for  determining  the  standards  of  performance  by  which 
to  assess  these  competencies.  (2)  The  instrument  was  designed  as  a 
behavi orally-anchored  rating  scale  to  determine  if  a  PT  student  can 
perform  clinical  competencies  necessary  for  entering  the  profession  as 
judged  by  four  standards  of  performance.  (3)  This  4-point  rating 
scale  was  tested  on  the  two  groups  of . PT  students  from  the  University 
of  Alberta.  The  ECC  was  administered  to  fourth  year  PT  students 
(j2=29 )  during  their  clinical  placement  in  November  1982  according  to 
the  specified  instructions  in  order  to  obtain  data  by  which  to  analyze 
and  revise  the  ECC.  The  revised  ECC  was  administered  to  third  year 
students  (_n=23 )  and  fourth  year  PT  students  (n=25)  during  their 
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clinical  placements  in  February  and  March  1983.  The  same  instructions 
were  followed  in  order  to  collect  data  necessary  to  establish  inter¬ 
rater  reliability,  concurrent  validity  and  construct  validity  of  the 
ECC. 

The  results  of  the  study  suggest  that  the  ECC  is  a  fairly 
reliable,  valid  and  practical  instrument  to  assess  clinical  competence 
of  PT  students.  The  inter-rater  reliability  of  the  ECC  was  .591  and 
.624  with  fourth  year  and  third  year  PT  students  respectively,  which 
surpassed  the  reliability  of  similar  clinical  evaluations  in  medicine 
and  PT.  The  ECC  discriminated  better  between  incompetent  and 
competent  levels  of  performance  than  between  competent  and  highly 
competent  levels  of  performance.  The  concurrent  validity  coefficients 
of  the  ECC  with  a  rating  of  potential  employabi 1 ity  were  .597  and  .655 
for  the  two  preceptors  assigned  to  evaluate  each  student.  Construct 
validity  was  supported  by  a  significant  difference  between  the  means 
of  the  total  ECC  scores  awarded  to  experienced  fourth  year  students 
and  novice  third  year  students  [F  (1,97)  =  26.06,  p  £  .001). 
Furthermore,  this  study  has  contributed  to  the  definition  of  the 
construct  of  clinical  competence  and  outlined  the  process  involved  in 
developing  a  clinical  evaluation  instrument. 
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Chapter  I 


INTRODUCTION 


Background  to  the  Problem 

Education  programs  in  the  health  professions  have  shifted 
emphasis  in  recent  years  toward  a  better  balance  between  the  academic 
and  clinical  components  of  the  program.  Evidence  is  that  clinical 
experience  now  occurs  much  earlier  in  the  students'  programs  in  all 
the  health  professions  and  there  is  greater  integration  between  the 
academic  and  clinical  components.  As  a  consequence,  the  need  for 
salient  evaluation  and  diagnosis  in  clinical  competence  can  not  be 
ignored.  The  purpose  of  this  present  study  was  to  develop  an 
instrument  for  evaluating  the  clinical  competence  of  physical  therapy 
(PT)  students  and  to  establish  its  reliability  and  validity. 

The  concept  of  competence  focuses  attention  on  the  completed 
behaviors  necessary  for  entering  professional  practice  rather  than  on 
the  curriculum  needed  to  develop  those  completed  behaviors  (Davis, 
Anderson  &  Jagger,  1979).  These  completed  behaviors  or  competencies 
are  the  knowledge,  skills,  judgements  and  attitudes  that  are  required 
for  adequate  performance  in  the  profession  (Berner  &  Bender,  1978). 

Although  the  PT  literature  dealing  with  clinical  competence  has 
become  more  extensive  in  recent  years,  it  has  focused  primarily  on 
defining  the  behaviors  leading  to  satisfactory  performance  (Ashton- 
McCrimmon  &  Hamel,  1983;  Chidley  &  Kisner,  1979;  Davis,  Anderson  & 
dagger,  1979;  May,  1977).  To  date,  few  clinical  evaluations  have  been 
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been  designed  to  assess  clinical  competence  of  PT  students  (Bemis, 
Smith  &  Mauser,  1978;  Texas  Consortium  for  Physical  Therapy  Clinical 
Education,  1981).  Furthermore,  limited  or  no  information  is  available 
concerning  their  reliability  and  validity.  The  narrative  method  of 
clinical  evaluation  remains  in  widespread  use,  despite  a  growing 
awareness  of  its  impreciseness  regarding  how  competent  a  student  may 
be  (May,  1977;  Mays,  1973). 

Clinical  evaluation  of  students  is  a  vital  component  to  the 
evaluation  process  in  the  PT  program  at  the  University  of  Alberta.  By 
completion  of  the  four  year  program  in  PT,  students  are  expected  to 
achieve  the  clinical  competence  required  for  adequate  performance  upon 
entering  the  profession.  To  assess  whether  a  student  is  ready  to 
practice  PT,  results  from  paper  and  pencil  tests  and  patient  simu¬ 
lations  in  the  classroom  setting  are  supplemented  by  narrative 
evaluations  of  student  performance  in  the  clinical  setting.  The 
narrative  evaluation,  however,  does  not  explicitly  assess  if  the 
desired  entry-level  clinical  competencies  have,  in  fact,  been 
achieved.  The  information  gathered  from  the  various  sources  is  used 
to  decide  whether  a  PT  student  will  be  promoted  to  the  next  year  of 
the  program  or  will  be  permitted  to  graduate. 

Statement  of  the  Problem 

Given  the  uses  of  clinical  evaluation  for  promotion  and  gra¬ 
duation,  a  reliable  and  valid  rating  instrument  to  assess  clinical 
competence  of  PT  students  should  be  developed.  Yet,  the  evaluation  of 
clinical  competence  is  a  complex  endeavor  for  the  behaviors  constitut¬ 
ing  a  competent  physical  therapist  are  not  fully  understood  (Ashton- 
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McCrimmon  &  Hamel,  1983).  Clearly,  an  evaluation  of  clinical 
competence  must  assess  behaviors  which  reflect  the  demands  of  the  job 
(Ashton-McCrimmon  &  Hamel,  1983;  Davis,  Anderson  &  Jagger,  1979;  May, 
1977).  Cri teri on-referenced  evaluation  provides  a  means  of  evaluating 
adequacy  of  an  individual's  performance  at  any  point  in  the  program 
against  specified  performance  standards  (Glaser  &  Nitko,  1971).  A 
cri terion-referenced  rating  scale  should  be  designed  to  include  the 
relevant  clinical  competencies  stated  in  behavioral  terms  to  depict 
what  a  student  must  do  to  meet  the  required  standards  of  performance. 
In  this  way,  evaluations  will  more  consistently  and  accurately  deter¬ 
mine  if  PT  students  have  acquired  the  requisite  knowledge,  skills, 
judgements  and  attitudes  necessary  for  effective  practice  as  physical 
therapi sts. 

Objectives  of  the  Study 

A  cri terion-referenced  evaluation  instrument,  "Evaluation  of 
Clinical  Competence"  (ECC),  was  developed  to  evaluate  clinical 
competence  of  fourth  year  PT  students  at  the  University  of  Alberta 
(UA).  The  steps  of  test  development  followed  were: 

1.  Clinical  competencies  required  for  PT  practice  were  identified 
and  validated,  and  the  standards  of  performance  by  which  to 
assess  the  competencies  were  determined. 

2.  The  instrument  was  designed  as  a  behavioral ly-anchored  rating 
scale  to  determine  if  a  PT  student  can  perform  clinical  com¬ 
petencies  required  for  entry  into  the  profession  as  judged  by 
four  standards  of  performance. 
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3.  The  4-point  rating  scale  was  tested  to  establish  its  reliability 
and  validity  with  fourth  year  PT  students  at  the  UA. 

Research  Questions 

The  instrument  development  phase  was  guided  by  two  questions: 

1.  What  are  the  behaviors  constituting  clinical  competence  in 
physical  therapy? 

2.  What  are  the  standards  by  which  to  judge  the  quality  of  perfor¬ 
mance  of  each  clinical  competency  included  in  the  evaluation 
instrument? 

The  instrument  validation  phase  asked  the  following  questions: 

1.  Is  the  inter-rater  reliability  of  the  ECC  equivalent  to  the 
reliability  measures  of  similar  clinical  evaluations  reported  in 
the  medical  and  physical  therapy  literature? 

2.  Do  students  who  rate  highly  on  the  ECC  also  have  high  ratings  as 
potential  employees? 

3.  Do  students  who  rate  highly  on  the  ECC  also  have  high  ratings  on 
the  narrative  evaluation  currently  in  use  at  the  UA? 

4.  Do  experienced  fourth  year  students  perform  better  than  novice 
third  year  students  on  the  ECC? 

5.  Do  students  and  preceptors  find  the  use  of  the  ECC  helpful  in 
the  professional  education  of  the  students? 

Significance  of  the  Study 

The  major  goal  of  the  PT  program  at  the  UA  is  to  assist  students 
to  develop  the  competencies  necessary  for  effective  practice  as  phy¬ 
sical  therapists  (May,  1977).  The  opportunity  for  the  student  to 
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acquire  these  desired  competencies  is  provided  by  both  the  academic 
and  clinical  components  of  the  program.  The  clinical  component  is 
valued  highly  by  the  Department  of  Physical  Therapy,  which  requires 
that  students  satisfactorily  complete  1200  hours  of  clinical 
experience  prior  to  graduation  (Policy  of  the  Department  of  Physical 
Therapy,  1981).  Yet,  no  research  has  been  conducted  on  the  methods  of 
evaluation  or  the  variables  of  the  clinical  training  environment. 

The  profession  of  PT  requires  its  practitioners  to  have  current 
knowledge,  evaluate  accurately,  execute  sound  judgements,  institute 
appropriate  treatment  and  display  proper  attitudes.  The  compassion 
with  which  patient  care  is  implemented  and  the  nature  of  the 
therapist-patient  relationship  are  also  aspects  of  the  therapist's 
role  (McDaniel,  1964).  In  today's  changing  health  care  system,  well 
prepared  physical  therapists  are  needed  to  assume  and  to  perform  more 
diverse  roles  and  responsibilities,  such  as  in  teaching,  supervising 
and  consulting,  which  in  the  past  had  not  normally  been  expected  from 
a  first-year  graduate  (Chidley  &  Kisner,  1979).  Ongoing  evaluation  of 
competencies  of  physical  therapists  in  the  clinical  setting  is  needed 
to  establish  a  firm  scientific  basis  from  which  to  determine  criteria 
for  measuring  clinical  competence  (Ashton-McCrimmon  &  Hamel,  1983). 

Observational  evaluation  provides  the  only  opportunity  to  assess 
certain  clinical  competencies,  hence  justifying  its  usefulness  in 
certifying  clinical  competence  (Irby,  Evans  &  Larson,  1978).  Narra¬ 
tive  reports  are  often  used  to  document  preceptors'  observations  of 
clinical  performance  of  PT  students  despite  their  susceptibility  to 
preceptor  biases  (Demers,  1978).  PT  preceptors  studied  by  Scully  and 
Shepard  (1983)  expressed  difficulty  in  recounting  and  writing  their 
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impressions  of  a  student's  behavior  and  progress  on  the  selected 
activities  included  in  the  report,  mainly  because  it  was  time- 
consuming  and  infringed  on  both  patient  care  and  personal  time.  Kapp 
(1981)  argued  that  preceptors  were  often  reluctant  to  evaluate 
candidly  due  to  an  apprehension  about  the  inescapable  subjective 
nature  of  evaluating  clinical  performance  and  a  fear  of  legal 
liability  accentuated  in  recent  years  by  societal  concerns  for 
students'  rights.  Despite  difficulties  and  concerns,  PT  preceptors 
felt  a  professional  responsibi 1 ity  to  evaluate  honestly  given  the  uses 
of  clinical  evaluation  in  PT  (Scully  &  Shepard,  1983).  In  addition  to 
the  hardships,  evaluation  had  a  positive  impact  on  professional  growth 
reminding  therapists  of  the  skills  that  they  should  possess  (Scully  & 
Shepard,  1983). 

Evaluations  of  students'  clinical  performance  must  be  both 

reliable  and  valid  if  they  are  to  be  worthwhile  for  making  decisions 
regarding  students'  clinical  competence-validity  being  limited  by  the 
extent  of  the  measures  of  reliability  (Allen  &  Yen,  1979).  Further¬ 
more,  the  increasing  demands  of  the  public  and  government  for 

accountability  of  health  professions  makes  the  development  of  reliable 
and  valid  measures  of  clinical  competence  in  PT  even  more  important 
(Newbie,  1976). 

Inaccurate  clinical  evaluations  fail  to  inform  the  students 
about  how  they  measure  up  in  terms  of  relevant  competencies  (Kapp, 
1981).  Furthermore,  the  students'  motivation,  confidence  and 

appreciation  of  specific  feedback  may  be  undermined  (Sternburg  & 
Brockway,  1979).  Equally  serious,  inaccurate  evaluation  may  promote 
or  graduate  students  when  they  should  not  be,  at  peril  to  the 
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profession,  the  public  and  the  student  (Kapp,  1981).  The  professional 
and  lay  public  have  an  interest  in  safeguarding  continued  competence 
by  ensuring  that  only  competent  students  will  graduate.  The  students, 
likewise,  have  a  vested  interest  in  their  career  development. 

Because  of  the  far-reaching  consequences  of  clinical  evaluation 
to  the  students,  the  public  and  the  profession,  every  attempt  should 
be  made  to  develop  a  well-designed  rating  instrument  in  PT  and  to 
establish  the  reliability  and  validity  of  its  results.  An  evaluation 
and  the  method  of  administration  should  be  guided  by  the  ultimate 
purpose  for  which  the  results  will  be  used,  should  be  congruent  with 
expectations  and  activities  of  those  being  evaluated,  and  should 
consider  the  effects  of  the  evaluation  process  on  the  individuals 
involved  (Berner  &  Bender,  1978). 

Del  imitations 

The  study  was  delimited  as  follows: 

1.  The  ECC  was  designed  to  evaluate  clinical  competence  of  fourth 
year  PT  students  only.  The  competencies  evaluated  are  congruent 
with  the  competencies  required  of  a  graduate  physical  therapist 
and  with  the  objectives  of  the  fourth  year  clinical  program. 
The  fourth  year  objectives  are  of  a  general  nature  stressing 
integration  of  competencies  and  problem-solving  skills  more  so 
than  in  the  third  year  program  where  acquisition  and  application 
of  specific  techniques  are  mainly  emphasized. 

The  study  was  limited  to  the  evaluation  of  clinical  competence 
of  PT  students  at  the  UA  during  their  compulsory  clinical 
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placements  at  clinical  facilities  in  Edmonton,  Calgary  and  Red 
Deer,  Alberta  and  Brockville,  Ontario. 

3.  Fourth  year  PT  student  data,  collected  during  two  different 

clinical  placements  (ri=29  and  rr=25,  respectively),  were  used  to 
evaluate  the  performance  of  the  ECC  with  fourth  year  PT  students. 

4.  Third  year  PT  student  data  collected  during  one  clinical 

placement  (n_=23)  were  used  to  verify  construct  validity  of  the 
ECC  intended  for  fourth  year  PT  students. 

5.  Two  preceptors  were  required  to  observe  and  evaluate  each 

student  for  15  minutes  daily  throughout  the  clinical  placement 
in  order  to  obtain  a  measure  of  inter-rater  reliability.  In  a 
typical  clinical  placement,  only  one  preceptor  supervises  and 
evaluates  student  performance. 

Limi tations 

The  following  are  considered  to  be  the  limitations  of  this 

research : 

1.  Students  were  volunteers  from  the  population  of  PT  students  in 
the  clinical  program  at  the  UA.  Only  three  fourth  year  PT 
students  did  not  volunteer  and  12  third  year  PT  students  did  not 
volunteer. 

i 

2.  Selection  of  preceptors  was  made  on  a  volunteer  basis  from  the 
population  of  physical  therapists  employed  by  the  clinical 
facilities  agreeing  to  participate  in  the  study. 

3.  The  sample  of  students  was  small  and  restricted  to  PT  students 
from  the  UA;  thus,  results  should  be  generalized  to  a  larger, 
similar  population  with  caution. 
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4.  One  of  the  preceptor  pairs  was  required  to  evaluate  students  on 
both  the  ECC  and  the  narrative  evaluation  currently  in  use  at 
the  UA;  therefore,  there  is  a  potential  source  of  carryover  from 
one  evaluation  to  the  other. 

5.  One  of  the  preceptor  pairs  was  also  responsible  for  supervision 
of  students  throughout  the  clinical  placement;  consequently, 
observations  of  students'  performance  outside  the  times 
scheduled  for  the  study  may  be  a  potential  source  of  bias. 

6.  Missing  data,  procured  when  the  "not  observed"  option  on  the 

rating  scale  was  used,  restricted  the  statistical  analysis. 

7.  The  necessity  for  two  preceptors  to  observe  and  evaluate  each 

student  induced  feelings  of  anxiety  in  the  students  which  might 
have  detracted  from  their  level  of  performance. 

8.  The  necessity  for  two  preceptors  imposed  an  extra  demand  on 

preceptors'  time  and  efforts  which,  if  it  infringed  on  patient 
care  responsibilities,  might  have  lessened  their  compliance  with 
the  requirements  of  the  study. 

Terms  and  Definitions 

The  following  terms  are  defined  as  they  are  key  terms  in  the 

text  of  the  thesis: 

clinical  competence  is  the  quality  of  being  capable  of  adequately 

performing  tasks  and  assuming  roles  of  a  physical  therapist  with 
their  associated  requisite  knowledge,  skills,  judgements  and 
attitudes. 

preceptors  are  physical  therapists  employed  by  clinical  facilities 

approved  to  supervise  and  evaluate  PT  students  from  UA  during 
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their  compulsory  clinical  placements, 
first  preceptor  is  the  physical  therapist  normally  assigned  to 
supervise  the  student  throughout  the  clinical  placement, 
second  preceptor  is  the  physical  therapist  working  in  close  proximity, 
who  is  assigned  to  observe  and  evaluate  the  student  for  the 
purposes  of  the  study. 

physical  therapists  are  graduates  of  an  accredited  program  of  physical 
therapy. 

clinical  placement  is  a  half-day  of  clinical  experience  for  PT  students 
in  approved  clinical  facilities  for  a  duration  of  four  weeks. 


Chapter  II 


SURVEY  OF  THE  LITERATURE 

There  are  many  reasons  cited  in  the  health  education  literature 
for  measuring  clinical  performance.  They  include:  (1)  certification 
of  competence,  (2)  maintenance  of  health  care  delivery,  (3)  provision 
of  feedback  to  the  students,  and  (4)  improvement  of  the  instruction 
process  (Mackenzie,  1974).  Fundamentally,  they  are  all  directed 
toward  producing  professionals,  whose  competence  can  be  trusted  by  the 
lay  and  professional  public. 

Methods  Used  in  Evaluating  Clinical  Competence 

Clinical  competence  is  defined  in  terms  of  either  the  outcomes 
of  clinical  performance  on  patient  care  or  in  terms  of  the  components 
of  the  process  leading  to  satisfactory  performance  (Barro,  1973; 
Newbie,  1976).  To  evaluate  both  defined  aspects  of  clinical 
competence,  different  methods  of  observation  and  measurement  must 
obviously  be  used. 

Evaluation  of  students  in  many  health  professions  is  conducted 
entirely  on  process  measures  as  students  rarely  assume  direct  respon¬ 
sibility  for  patient  care,  an  essential  prerequisite  for  outcome 
evaluation.  Process  evaluations  are  based  on  what  a  practitioner 
actually  does  to  provide  proficient  and  safe  care  to  patients  (Newbie, 
1976). 


11 


— ~ 


12 


Factor  analysis  of  medical  evaluations  has  revealed  a  number  of 
factors  of  the  construct  of  clinical  competence.  These  factors  may 
assist  in  the  determination  of  criteria  constituting  successful 
performance.  DiMatteo  and  DiNicola  (1981)  identified  two  factors 
termed  technical  skills  and  interpersonal  skills  which  accounted  for 
27.1%  and  60.4%  of  the  variance,  respectively.  Similarily,  Davidge, 
Davis  and  Hull  (1980)  found  two  factors  which  they  called  problem¬ 
solving  and  interpersonal  skills.  Dowaliby  and  Andrews  (1976) 
identified  three  factors:  data  gathering,  interpersonal  skills  and 
clinical  judgement.  Anwar,  Bosk  and  Greenburg  (1981)  classified  the 
many  factors  found  after  separate  analysis  of  thirty-eight  surgical 
evaluation  forms  into  seven  categories  of  clinical  competence. 
Knowledge,  clinical  skills  and  clinical  judgement  comprised  31.8%  of 
the  total  factors.  Intellectual  capacity  (teaching  and  oral  communic¬ 
ation)  and  work  habits  accounted  for  23.9%  of  all  the  factors. 
Personal  characteristics,  such  as  dependability,  reliability,  respon¬ 
sibility,  emotional  stability,  maturity,  integrity,  honesty,  morality 
and  motivation,  accounted  for  another  27%.  Interpersonal  relations 
made  up  the  remaining  18%.  In  summary,  clinical  competence  is  the 
knowledge,  skills,  judgements,  and  attitudes  required  for  adequate 
performance  for  delivery  of  optimal  patient  care  (Berner  and  Bender, 
1978). 

The  variety  of  technical  and  interpersonal  skills  to  be 
evaluated  in  health  education  demands  that  a  wide  variety  of  process 
evaluation  methods  be  used.  Multiple-choice  tests  are  frequently 
used.  They  are  very  reliable,  easy  to  score,  and  cover  large  areas  of 
content;  however,  they  tend  to  measure  predominantly  recall  of 
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knowledge  (Levine,  McGuire  and  Nattress,  1970).  Newbie  (1976)  argued 
that  examinations  to  certify  clinical  competence  should  not  rely 
solely  on  multiple-choice  questions  but  rather  should  emphasize 
application  of  knowledge  to  the  clinical  situation.  Other  more  valid 
measures  should,  therefore,  be  included:  patient-management  problems, 
patient  simulations,  computer  simulations,  cri terion-referenced 
clinical  examinations,  oral  examinations,  direct  observation  and 
multiple  methods  (Newbie,  1976). 

Observational  Evaluations 

Direct  observation  entails  systematic  observation  of  the  inter¬ 
action  between  students  and  their  patients,  followed  by  an  evaluation 
of  "on-the-job  performance"  using  checklists  and  rating  scales. 
Factors  suggested  by  Levine  (1978)  which  hamper  the  use  of  the  rating 
scale  as  an  objective  measurement  instrument  are  the  poorly  designed 
evaluation  instruments  in  use,  the  complexity  of  behavior  to  be 
evaluated,  the  lack  of  well  developed  competencies,  the  lack  of 
training  for  evaluators  in  the  use  of  the  instrument  and  rater  biases 
coupled  with  insufficient  time  for  observation  and  the  diversity  of 
the  clinical  environment.  However,  the  educational  measurement 
literature,  by  virtue  of  its  definition  of  evaluation,  accepts  that 
the  evaluation  process  can  never  be  fully  objective.  Evaluation  is 
the  qualitative  and  quantitive  descriptions  of  behavior  along  with 
descriptions  of  value  judgements  concerning  the  desirability  of  the 
behavior  (Gronlund,  1976). 

Irby,  Evans  and  Larson  (1978)  stated  that  observational  evalua¬ 
tion  as  determined  by  rating  scales  is  a  justifiable  method  by  which 
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to  certify  competence,  despite  its  recognized  subjectivity,  because  it 
provides  the  only  opportunity  to  assess  certain  competencies.  The 
competencies  that  can  be  observed  in  a  clinical  setting  include  not 
only  the  students'  applied  knowledge  and  technical  skills  but  also 
students'  clinical  judgements,  interests,  work  habits,  communication 
and  interpersonal  skills  (Anwar,  Bosk  and  Greenburg,  1981).  Wigton 
(1980)  used  the  Delphi  technique  to  rank  variables  considered 
important  to  performance  of  first  year  resident  physicians.  The  four 
highest  ranking  qualities  were  dependability,  relationship  with 
patients,  clinical  judgement  and  problem  formulation.  Only  the  latter 
lends  itself  to  objective  evaluation.  Bosk  (1979)  indicated  that  in 
surgical  training  there  was  also  strong  emphasis  placed  on  qualities 
that  exemplify  honesty,  trustworthiness,  ethics  and  dependability 
which  required  a  subjective  i nterpretation  of  the  motives  and  morals 
of  the  surgical  resident's  professional  conduct. 

Even  though  observational  evaluative  methods  offer  the  most 
realistic  appraisal  situation,  this  advantage  simultaneously 
interferes  with  standardization  of  the  evaluation  process.  Each 
student  will  be  seen  with  different  patients  of  varying  complexity 
which  can  lead  to  environmental  errors,  and  each  student  will  be  seen 
from  the  observer's  own  point  of  view  leading  to  observer  errors. 

The  subjectivity  of  observational  evaluations  stems  mainly  from 
errors  introduced  by  the  personal  biases  of  the  observers,  particu¬ 
larly  when  evaluating  covert  behavior  or  that  which  is  not  readily 
observable  (DeMers,  1978).  Guilford  (1954)  identified  six  common  bias 
errors,  some  of  which  have  been  studied  by  the  health  professions: 
(1)  error  of  leniency/stringency ,  (2)  error  of  central  tendency,  (3) 
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halo  effects,  (4)  logical  error,  (5)  proximity  error,  and  (6)  contrast 
error. 

Errors  in  Observational  Evaluations 

Petersdorf  (1977)  reported  stringency  error  and  leniency  error 
among  oral  examiners  in  medicine.  The  stringent  examiners  had  a 
failure  rate  as  high  as  53%;  lenient  examiners  had  a  failure  rate  as 
low  as  7%.  Waugh  and  Moyse  (1969)  observed  during  oral  examinations 
that  some  examiners  pressed  for  preconceived  responses  and  others  did 
not  encourage  students  who  were  not  doing  well.  Dietz  and  Slaymaker 
(1976)  attributed  the  clustering  of  Occupational  Therapy  field 
performance  scores  toward  the  high  end  of  the  scale  to  a  leniency 
error  whereby  raters  consistently  gave  students  the  benefit  of  the 
doubt.  The  central  tendency  may  have  also  operated  here,  in  that 
raters  tend  to  avoid  low  ratings  to  avoid  justifying  the  low  rating  or 
to  avoid  appearing  as  a  poor  teacher. 

Bull  (1959)  suggested  that  factors  related  to  personality  play  a 
more  significant  role  in  determining  clinical  grades  in  medicine  than 
knowledge  and  skill.  Marienfield  and  Reid  (1980)  demonstrated  a  halo 
effect  where  preceptors  overestimated  knowledge  for  medical  students 
described  as  highly  motivated  and  attentive  to  patient  care.  Fourteen 
students  whose  knowledge  was  rated  as  superior  by  four  to  five 
preceptors  performed  only  marginally  on  the  National  Board  of  Medical 
Examiners  (NBME)  internal  medicine  subtest  (n=188).  Beneson,  Stimmel 
and  Aufses  (1981)  also  illustrated  a  false  negative  error  where  61%  of 
the  students  who  were  classified  as  honors  by  preceptors  on  a  surgical 
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clerkship  (n=51)  were  below  the  cutting  score  for  honors  classifica¬ 
tion  on  the  NBME  surgical  subtest. 

Anwar,  Bosk  and  Greenburg  (1981)  found  that  a  surgical  resident 
earned  discredit  if  errors  made  were  repeatedly  viewed  as  blame-worthy 
or  of  the  resident's  own  fault;  a  surgical  resident  earned  credit  if 
errors  made  were  seen  as  blameless  or  of  no  fault  of  the  resident. 
They  revealed  that  poor  knowledge,  skills  and  judgement  were  given  as 
reasons  in  75%  of  those  residents  not  advanced  to  chief  resident 
and/or  not  certified  for  the  NBME;  and  problems  with  personal 
qualities  were  the  reasons  given  for  dismissal  of  23%  of  those 
residents  not  advanced  and/or  not  certified.  They  concluded  that  poor 
clinical  judgement  is  not  a  reason  for  dismissal  unless  coupled  with  a 
deficit  in  personal  qualities.  Scully  and  Shepard  (1983)  disclosed 
that  physical  therapists  supervising  students  tolerated  deficiencies 
in  knowledge  and  performance  skills,  expressing  optimism  that  perfor¬ 
mance  difficulties  can  be  overcome  with  increased  experience. 
Deficiencies  in  the  ability  "to  work  with  people",  however,  were  less 
tolerated  expressing  that  affective  skills  are  influenced  by  life 
experiences  and  are  difficult  to  change.  Holloway,  Collins  and  Start 
(1968)  reported  that  extroverted  students  received  higher  ratings  than 
warranted  in  oral  examinations,  while  introverted  students  obtained 
lower  ratings  than  called  for. 

Wigton  (1981)  provides  a  good  example  of  the  variation  in 
assessment  which  arises  from  personal  qualities.  Twenty-five  video¬ 
taped  case  histories  were  prepared  by  five  different  students 
presenting  five  case  histories  which  varied  in  degree  of  clinical 
sophistication.  Each  of  the  15  faculty  members  ranked  five  presenta- 
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tions  (1  best,  5  worst)  and  each  presentation  was  ranked  by  three 
evaluators.  There  was  a  significant  difference  in  rank  given  depend¬ 
ing  both  on  which  student  was  presenting  (p  ^  .006)  and  what  case  was 
being  presented  (p  <  .003).  Students  A  and  C  ranked  higher  than 

students  B  and  D,  who  ranked  higher  than  student  E.  Case  2,  an 
unsophisticated  but  thorough  history,  written  by  a  medical  student  on 
his  first  clerkship,  ranked  above  the  other  cases  written  by  senior 
medical  faculty,  staff  and  students.  Case  2  was  judged  best  when 
presented  by  student  A  and  worst  when  presented  by  student  E.  The 
correlation  coefficient  between  any  two  faculty  members  viewing  the 
identical  presentation  was  .39  when  standard  scores  were  used  to 
correct  for  evaluator  stringency  or  leniency  errors. 

Given  that  subjectivity  is  intrinsic  to  clinical  evaluation,  the 
question  that  still  remains  is  how  can  evaluation  be  designed  so  that 
both  public  and  professional  interests  are  served.  This  statement 

implies  that  evaluations  must  be  designed  to  reduce  uncertainty  and 
bias,  thereby  providing  more  consistent  or  reliable  results.  A  number 
of  methods  are  suggested  in  the  medical  education  literature  to 
improve  rel iabil ity. 

Methods  to  Improve  Reliability  of  Rating  Scales 

Reliability  is  essential  to  rating  scales  because  it  provides 

more  consistent  and  accurate  results  (Anwar,  Bosk  and  Greenburg, 

1981).  Rating  scales  for  resident  physicians  and  medical  students 
have  been  most  extensively  studied  and  have  been  found  to  be  quite 
reliable  with  measures  of  internal  consistency  ranging,  for  the  most 
part,  from  .20  to  .85.  According  to  the  classification  of  observer 
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agreement  for  categorical  data  suggested  by  Landis  and  Koch  (1977),  a 
Kappa  coefficient  between  .21  and  .40  is  a  fair  level  of  agreement; 
whereas,  0  to  .20  is  only  slight  agreement  and  .41  to  .60  is  moderate 
agreement  between  observers.  Low  reliabilities  indicate  that 
differences  in  performance  by  individual  students  is  mainly  due  to 
rater  error  and  variable  clinical  situations.  Reliability  of  a  rating 
scale  can  be  increased  by  encorporating  factors  during  its  construc¬ 
tion  which  reduce  the  amount  of  rater  error  (Levine,  1978).  These 
factors  are  discussed  next. 

Well-defined  Competencies  and  Performance  Criteria 

The  competencies  necessary  for  entry  into  a  profession  must  be 
identified,  organized  in  order  of  importance  and  written  in  appro¬ 
priate  terminology  (Berner  and  Bender,  1978).  In  search  of  clinically 
significant  competencies,  they  advised  a  review  of  the  literature  and 
professional  documents  and  a  review  of  clinical  practice.  Methods 
recommended  to  obtain  data  directly  from  clinical  practice  were:  (1) 
chart  audit,  (2)  critical  incidence  technique  as  proposed  by  Flanagan, 
(3)  diary  study  of  practitioners,  and  (4)  task  analysis  by  expert 
judges. 

After  competencies  have  been  identified,  they  should  be  worded 
in  a  way  that  tells  both  the  student  and  evaluator  what  the  student 
must  do,  in  terms  of  observable  behavior,  to  perform  acceptably 
(Shepard,  1977).  Baker  (1974)  suggested  that  competencies  be  defined 
in  terms  of  domains.  Some  competencies  require  the  student  to 
demonstrate  performance  skills  (psychomotor  domain),  others  require 
the  student  to  have  knowledge  (cognitive  domain),  while  still  others 
involve  attitudes  (affective  domain).  Taxonomies  are  available  to 
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help  decide  on  a  level  of  performance  to  be  expected  within  each 
domain  (Campbell,  1977). 

Associated  with  each  competency  is  a  rating  scale  which  allows 
the  observer  to  rate  the  quality  of  performance  seen  by  comparing  it 
against  the  predetermined  criteria  for  successful  performance.  The 
criteria,  also  described  in  terms  of  observable  behavior,  are 
illustrative  of  the  levels  of  mastery  of  competence. 

Dielman,  Hull  and  Davis  (1977)  reported  inter-rater  reliability 
statistics  on  a  behavioral ly-anchored  rating  scale  of  on-the-job 
performance  which  defined  four  levels  of  behavior  under  each  of  the 
fifteen  items  in  the  scale.  Over  one  year  there  were  1749  ratings  of 
355  medical  students  completed  by  resident  physicians  and  1908  ratings 
of  389  medical  students  completed  by  attending  physicians.  The 
inter-rater  reliability  for  residents'  ratings  for  the  total  scale  was 
.61  ranging  from  .30  to  .51  for  the  fifteen  single  items;  the  inter¬ 
rater  reliability  for  attendings'  ratings  for  the  total  scale  was  .40 
ranging  from  .22  to  .37.  The  higher  reliabilities  for  residents  may 
be  because  they  showed  more  contact  hours  with  students  than  did 
attending  physicians  (t=8.62  p  ^  .001)  (Erviti,  Fabrey  and  Bunce, 
1979). 

Inter-rater  reliabilities  are  somewhat  lower  for  the  more 
subjectively  worded  rating  scales  of  clinical  performance  than  the 
behavioral ly-anchored  rating  scales.  Reliabilities  of  a  subjective 
rating  scale  reported  by  Littlefield  (1981)  was  low  (.13)  during  the 
first  year  of  use  with  medical  students  but  greatly  improved  (.42) 
over  four  years,  presumably  due  to  the  annual  feedback  that  was 
provided  to  the  faculty.  The  coefficient  increased  from  .42  to  .65  in 
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the  fourth  year  when  the  number  of  preceptors  per  student  was 
increased  from  two  to  five  with  the  addition  of  residents  as 
preceptors.  The  mean  of  the  resident  rating  was  significantly  higher 
(9.45)  than  the  mean  faculty  rating  (8.79).  O'Donohue  and  Wergins 
(1978)  reported  similar  reliabilities  of  ratings  of  medical  students 
with  intraclass  correlations  of  .381  for  residents  and  .323  for 
attending  physicians.  Residents  also  tended  to  give  slightly  higher 
mean  scores  than  attendings,  but  the  ratings  correlated  with  each 
other  (r=.704),  indicating  that  an  average  of  multiple  ratings  by 
these  two  groups  was  more  consistent  than  individual  ratings  for  each 
student.  This  finding  is  consistent  with  the  Spearman-Brown  formula. 

Furthermore,  in  dentistry  Gaines,  Bruggers  and  Rasmussen  (1974) 
compared  effects  of  subjectively  and  objectively  worded  performance 
criteria.  They  showed  inter-rater  reliability,  as  estimated  by  intra¬ 
class  correlation  coefficients,  increased  from  .26  with  subjective 
criteria  to  .56  with  objective  criteria  defining  a  3-point  scale  for 
the  seven  raters  evaluating  wax  carvings  of  eight  students  on  two 
occasions  three  months  apart.  Analysis  of  variance  showed  significant 
differences  between  raters  when  using  the  subjective  scale  but  no 
significant  differences  when  using  the  objective  scale. 

In  contrast  to  the  high  reliabilities  for  resident  physicians  in 
the  previous  studies  mentioned,  examination  of  resident  performance 
evaluations  from  four  different  sources  by  DiMatteo  and  DiNicola 
(1981)  found  that  the  highest  reliabilities  were  among  ratings  by 
attending  physicians  with  alpha  coefficients  ranging  from  .83  to  .96 
followed  next  by  residents  themselves  with  coefficients  ranging  from 
.67  to  .93.  Self  ratings  were  the  least  internally  consistent  (.56  to 
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.83),  while  patient  ratings  were  nearly  as  reliable  as  both  the 
attending  and  resident  physician  ratings.  Oaks,  Scheinok  and  Husted 
(1969)  also  revealed  that  residents  and  full  professors  assigned 
significantly  more  inaccurate  ratings  to  medical  students  than 
associate  and  assistant  professors  using  the  Goodman  procedure  of 
mismatches.  They  concluded  that  residents,  aespite  close  contact  with 
students,  may  be  vulnerable  to  personal  biases.  Further  to  this, 
Erviti,  Febrey  and  Bunce  (1979)  did  not  find  a  systematic  difference 
between  the  reliability  of  residents  and  faculty  evaluating  medical 
students . 

Scales  with  highly  specific  competencies  and  criteria  yield 

higher  reliability,  particularly  when  the  scales  are  task-specific. 
Lui  (1980)  found  that  inter-rater  reliability  was  .75  for  a  spinal 

anaesthesia  skill  and  .83  for  anaesthesia  setup  and  machine  checkout. 
Fiel,  Griffen,  McNeil,  Ajunwa,  Salisbury  and  Aasved  (1979)  demon¬ 

strated  an  inter-rater  reliability  of  .87  for  a  post  test  measuring 
blood  pressure  readings.  Schor,  Grayson,  Nugent  and  Oken  (1977) 

reported  that  a  simulation  evaluation  with  students  in  a  Health 
Associate  program  yielded  alpha  coefficients  of  .86  for  data 

gathering,  .97  for  physical  examination,  .94  for  interpersonal  skills 

and  .88  for  health  education.  Improved  reliabilities  among 

task-related  rating  scales  illustrate  that  whenever  possible 
evaluation  should  be  limited  to  behavior  as  it  is  occurring. 

The  findings  by  Schor,  et  al.  (1977)  coincide  with  findings  by 
DiMatteo  and  DiNicola  (1981)  where  items  measuring  i nterpersonal  skill 
of  resident  physicians  had  higher  reliabilities  (.44  to  .96  with  a 

median  coefficient  of  .90)  than  did  items  measuring  technical  skill 
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(.36  to  .91  with  a  median  coefficient  of  .79).  Liston,  Yager  and 
Strauss  (1981)  also  found  agreement  by  psychi atri sts  was  significantly 
poorer  for  judgements  about  technique  skills  than  about  communication 
skills  with  psychiatry  residents  (X^=8.57  p  ^  .01).  Scherer  and 
Rogers  (1980)  disclosed  that  nonverbal  cues  of  immediacy  (close 

therapist-client  distance  and  eye  contact)  significantly  improved 

ratings  of  .the  therapist's  interpersonal  skills  and  effectiveness 
(F=50.69  p  £  .0001).  These  results  may  indicate  that  there  is  greater 
uniformity  among  the  components  of  interpersonal  skill  than  among  the 
components  of  technical  skill.  However,  the  ratings  of  residents' 

interpersonal  and  technical  skills  by  attending  physicians  were 
substantially  correlated  with  one  another  (r=.74)  (DiMatteo  and 

DiNicola,  1981),  which  may  reflect  a  strong  relationship  between  both 
aspects  of  care  in  actual  medical  practice. 

Number  of  Scale  Points 

DeMers  (1978)  stated  that  three  to  seven  scale  points  should  be 
provided;  more  points  should  be  used  when  fine  discriminations  can  be 
made.  Also,  some  provision  should  be  made  on  the  scale  to  permit 
evaluators  to  omit  items  that  they  feel  unqualified  to  judge. 

There  is  evidence  to  suggest  that  employing  the  minimum  number 
of  useful  points  increases  the  reliability.  Erviti,  Fabrey  and  Bunce 
(1979)  found  that  the  average  Kappa  coefficient  over  the  forty 
resident  and  faculty  pairs  was  .26,  but  when  the  4-point  scale  was 
reduced  to  a  2-point  scale  across  all  forty  pairs,  the  reliability 
coefficient  increased  to  .53.  The  mean  number  of  times  that  the  "do 
not  know"  option  was  used  was  significantly  less  for  residents  than 
faculty  (t=5.93  p  ^  .001).  This  result  seems  reasonable  considering 
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that  the  number  of  student  contact  hours  was  significantly  higher  for 
residents,  as  stated  earlier. 

In  dentistry,  Houpt  and  Kress  (1973)  compared  a  2-point  scale  to 
a  5-point  scale  for  evaluating  cavity  preparations  on  eight  criteria. 
Both  intra-rater  and  inter-rater  reliabilities  were  higher  when  using 
the  2-point  scale  to  evaluate  the  eight  criteria,  but  reliability  for 
the  overall  scores  did  not  differ  significantly  for  the  two  scales. 

Multiple  Observations 

Inter-rater  reliability  seems  to  be  low  when  dealing  with 
judgements  from  individual  observers,  therefore,  ratings  from  several 
observers  should  be  collected.  The  reliability  of  ratings  on  single 
items  can  increase  when  ratings  by  several  observers  are  averaged, 
thus  eliminating  the  error  due  to  observer  variation  (Printen, 
Chappell  and  Whitney,  1973).  Further  to  this,  they  reported  that  the 
estimated  correlation  between  clinical  performance  ratings  by  two 
raters  averaged  across  the  ten  item  rating  scale  was  .64,  while  the 
probability  of  a  student  receiving  the  same  score  from  two  separate 
groups  of  six  raters  was  calculated  to  be  .92. 

DiMatteo  and  DiNicola  (1981)  recommended  that  resident  perfor¬ 
mance  evaluations  be  conducted  by  ratings  from  attending  physicians, 
peers,  patients  and  self-ratings.  The  substantial  reliabilities  from 
all  four  sources  demonstrated  the  accuracy  and  utility  of  these 
sources  of  physician  evaluation.  However,  the  intercorrel ations  among 
evaluation  from  these  sources  were  also  fairly  independent,  indicating 
that  they  provide  separate  measures  of  physician  performance.  Attend¬ 
ing  physicians  and  peers  tended  to  agree  best  with  each  other  (.64). 
Self-ratings  had  a  moderate  relationship  to  ratings  by  attendings 
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(.31)  and  peers  (.38).  Patient  ratings  shared  little  variance  with 
ratings  by  the  attendings  (.16),  peers  (.02)  and  self  (.10). 

Another  approach  to  multiple  observation  is  use  of  multiple 
methods  whereby  a  battery  of  methods  attempt  to  match  the  most  appro¬ 
priate  method  to  the  aspects  of  clinical  competence  being  tested. 
Systematic  study  of  evaluation  methods  is  needed  to  provide  more 
comparative  data  for  enhancement  of  the  study  of  clinical  evaluation. 

O'Donohue  and  Wergins  (1978)  utilized  preceptor  ratings  of 
clinical  performance,  oral  examinations  and  written  examinations  to 
evaluate  medical  students  in  a  clerkship.  The  reliability  of 
preceptor  ratings  was  .323  for  attending  physicians,  in  contrast  to 
the  reliability  of  oral  examiners  at  .754  (p  6  .001).  The  inter¬ 
correlations  of  the  three  methods  were  small,  ranging  from  .19  to  .25, 
indicating  that  each  evaluation  assessed  different  aspects  of  student 
performance.  Littlefield  (1981)  also  utilized  the  same  three  evalua¬ 
tion  methods  with  similar  results.  The  reliability  of  preceptor 
ratings  was  .42  for  attendings.  The  oral  (.78  to  .92  over  four  years) 
and  written  (.59  to  .84  over  four  years)  examinations  were  highly 
reliable.  Again,  modest  intercorrelations  suggested  that  multi -method 
evaluations  capture  distinct  aspects  of  performance. 

Methods  to  Improve  Validity  of  Rating  Scales 

Reliability  is  an  important  requirement  of  clinical  evaluation 
because  it  limits  validity,  but  reliability  aoes  not  guarantee 
validity  or  relevancy  of  the  qualities  measured  (Allen  and  Yen, 
1979).  Anwar,  Bosk  and  Greenburg  (1981)  warned  that  there  is  a  danger 
in  assuming  that  those  qualities  which  can  be  easily  and  reliably 
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measured  are  the  only  ones  relevant  to  evaluation. 

To  be  sure  that  a  rating  scale  evaluates  students  on  what  they 
are  supposed  to  do  in  order  to  be  ready  to  practice  a  profession,  the 
steps  of  test  development  recommended  by  Allen  and  Yen  (1979)  should 
be  followed.  These  steps  are:  (1)  plan  a  test  blueprint  to  cover  all 
the  important  competency  areas,  (2)  write  items  for  each  competency 
area  in  the  plan,  (3)  administer  all  the  items  to  examinees  represent¬ 
ative  of  the  population  with  which  the  final  test  will  be  used 

according  to  standardized  instructions  and  time  limits,  (4)  select  the 
best  items  and  refine  them  if  necessary,  and  (5)  administer  the 
revised  test  on  another  representative  sample  of  examinees  under  the 
same  standardized  conditions. 

In  the  planning  stage  (steps  1  and  2),  crucial  to  content 
validity,  a  test  blueprint  is  developed  so  that  all  the  important 

competencies  are  sampled.  Weights  may  be  given  to  certain  competen¬ 
cies  to  indicate  their  degree  of  importance  relative  to  the  other 
competencies  sampled  (DiMatteo  and  DiNicola,  1981;  Oaks,  Scheinok  and 
Husted,  1969).  Weights  are  particularly  helpful  when  important 
competencies  are  not  readily  amenable  to  observation.  The  validation 
stage  (steps  3  to  5)  may  be  designed  to  test  the  three  types  of 

validity:  concurrent,  predictive  and  construct  validity. 

Little  research  has  been  done  on  construct  validity.  Lass, 
Kornreich,  Hoffman  and  Friedman  (1977)  demonstrated  consistency  and  an 
upward  trend  in  ratings  of  clinical  performance  of  the  same  students 
throughout  medical  school  and  internship.  Davidge,  Davis  and  Hull 

(1981)  found  that  students  who  were  evaluated  both  early  and  late  in 
the  clerkship  showed  significant  improvement  on  problem-solving  items 
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but  did  not  show  improvement  in  interpersonal  skill  items.  These 
results  may  be  taken  as  evidence  supporting  the  construct  validity  of 
the  rating  scale,  inasmuch  as  the  goals  of  the  clerkship  emphasize 
improvements  in  problem-solving  skills  over  interpersonal  skills. 
Furthermore,  improvement  is  expected  as  more  experience  is  gained  from 
early  to  later  clerkships  during  the  training  program. 


Development  and  Validation  of  Physical  Therapy  Competencies 

A  number  of  articles  and  professional  documents  have  been 
written  in  order  to  upgrade  clinical  curriculum  and  supervision  in 
PT.  Yet,  the  underlying  constructs  or  competencies  of  good  PT 
practice  have  not  been  well  delineated.  An  understanding  of  the 
construct  would  enhance  evaluation  and  differentiation  between  "good" 
and  "poor"  PT  students. 

Analysis  of  current  professional  practice  would  identify 
competencies  which  would  also  serve  as  a  basis  for  establishing 
performance  standards  or  criteria.  McDaniel  (1964)  applied  the 
critical  incident  technique  as  a  method  of  objectively  finding  actions 
that  distinguish  between  good  and  poor  physical  therapists.  Four 
hundred  ninety  four  incidents  reported  by  supervisors  of  PT  depart¬ 
ments  across  the  Uni  tea  States  were  analyzed.  A  profile  of  an  ideal 
physical  therapist  was  described  by  McDaniel  (1964)  as  one  whom: 


related  well  to  patients  by  developing  confidence  and 
interest;  made  good  clinical  judgements  on  treatment 
selection,  progression,  patient  referrals,  appliance 
selection  and  patient  safety;  interpreted  and  carried  out 
orders  wisely,  improvising  as  needed;  adjusted  patient 
treatment  as  physical  and  psychological  needs  varied; 
taught  patients  and  family  effectively;  coordinated  closely 
with  other  services  and  handled  equipment  with  care, 
(p.  241) 
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Hence,  clinical  competence  is  something  more  than  knowledge  and  skill, 
it  also  involves  clinical  judgement,  which  is  reflected  by  the  ability 
to  improvise  and  to  solve  problems  (Watts,  1971). 

Ashton-McCrimmon  and  Hamel  (1983)  compiled  a  list  of  236  com¬ 
petencies  through  use  of  documentation  descriptive  of  PT  practice. 
These  competencies  were  subsequently  edited  to  224  competencies  by  six 
PT  experts..  The  importance  of  these  competencies  was  rated  on  a 
questionnaire  sent  to  349  practicing  physical  therapists  in  Quebec. 
Importance  ratings  of  the  competencies  were  given  by  179  respondents. 
Competencies  were  grouped  into  eleven  categories.  The  importance 
rating  of  the  categories  ranked  according  to  average  mean  results  from 
the  most  important  to  the  least  important  were:  planning  of  treatment 
(78.8%  importance  score),  theoretical  knowledge  (73.3%),  interpersonal 
relations  and  communication  skills  (73.3%),  evaluation  (71.3%), 
personal  qualities  (64.1%),  professional  ethics  and  attitudes  (69.4%), 
professional  growth  (68.1%),  administrative  skills  (51.1%),  awareness 
of  health  delivery  (45.1%),  treatment  skills  and  implementation 
(41.6%)  and  research  skills  (27.5%).  There  were  significant 
differences  in  importance  ratings  for  years  of  experience  and  place  of 
training.  Physical  therapists  with  up  to  two  years  of  experience  and 
those  with  more  than  fifteen  years  of  experience  both  rated  the 
categories  higher  than  did  the  other  five  groups  with  intermediate 
years  of  experience.  The  group  with  more  than  fifteen  years  of 
experience  gave  the  highest  importance  ratings  to  professional  ethics, 
i nterpersonal  skills,  awareness  of  health  delivery  and  administrative 
skills;  whereas,  the  group  with  up  to  two  years  of  experience  had 
higher  ratings  on  evaluation,  treatment  implementation  and  research 
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skills.  Forei gn-trained  therapists  rated  all  categories  higher  than 
did  those  trained  in  Quebec. 

Wilhelm  (1969)  used  the  Q  sort  technique  to  identify  tasks  that 
PT  students  must  do  successfully.  Student  behavior  was  described  on 
two  dimensions:  a  competency  domain  and  the  level  of  competency.  Ten 
competency  domains  were  found.  Three  levels  of  competency  defined 
each  domain..  The  levels  of  competence  began  with  the  ability  to  apply 
knowledge  under  supervision,  progressed  to  the  ability  to  apply  know¬ 
ledge  to  new  situations  when  supervised  and  evolved  to  the  ability  to 
practice  without  direct  supervision.  A  rating  scale  was  constructed 
using  these  two  dimensions.  No  validation  study  was  done  to  verify 
either  competencies  identified  or  reliability  and  validity  of  the 
instrument  itself. 

The  Ohio  University  Physical  Therapy  Department  (Chialey  and 
Kisner,  1979)  used  expert  judges  to  conduct  a  task  analysis  of 

clinical  education  and  define  the  competencies  involved  in  competent 

performance.  Clinical  education  was  characterized  by  three  program 
goals:  direct  patient  care,  PT  delivery  and  personal/professional 

growth.  Under  each  goal,  seven  entry-level  competencies  for  PT 

graduates  were  defined  as  evaluating,  planning,  communicating, 
treating,  teaching,  supervising,  and  contributing.  This  list  of 
entry-level  competencies  was  validated  through  use  of  a  questionnaire 
sent  to  all  1974  and  1975  graduates  of  the  program.  Interesti ngly, 
the  results  indicated  a  high  demand  for  such  functions  as  evaluating, 
teaching  and  supervising,  which  had  not  been  stressed  in  earlier 
studies  (McDaniel,  1964;  Wilhelm,  1969).  Competence  was  conceptua- 


. 


1 


29 


lized  as  adequacy  of  physical  therapist  performance  and  patient 
service  emerging  from  competencies  in  all  three  program  goals. 

The  Canadian  Physiotherapy  Association  in  1982  developed  the 
Recommended  Core  Curriculum  for  Physiotherapy  Education  Programmes 

which  outlined  what  a  student  is  expected  to  perform  upon  graduation. 
The  section  on  clinical  practice  was  divided  into  four  content  areas 
called  therapeutic  process,  communication,  documentation  and  safety 
under  which  were  listed  the  criteria  of  performance. 

The  American  Physical  Therapy  Association  in  1981  developed  a 
manual ,  Competencies  in  Physical  Therapy:  An  analysis  of  practice, 
following  a  task  analysis  of  practicing  physical  therapists.  The 
operational  definition  of  competency  was  "a  significant  behavior  or 
activity,  performed  in  a  specific  setting,  to  a  specific  standard"  (p. 
1088).  For  each  behavior  or  activity  a  major  competency  statement  was 
made  with  necessary  situational  givens  and  standards.  The  activity 
was  also  depicted  in  a  Gagne's  heirarchical  array  of  terminal  com¬ 
petencies  with  their  supporting  subactivities,  skills  and  knowledge 
(Davis,  Anderson  &  Jagger,  1979). 

In  summary,  the  analysis  of  clinical  competence  into  discrete 
content  areas  may  serve  as  tentative  parameters  for  defining  the 
construct  of  clinical  competence.  It  should  be  possible  to  make 
generalizations  applicable  to  students  from  studies  of  practitioners. 
An  analysis  of  competence  informs  supervising  therapists  on  what  to 
expect  from  students  and  guides  students  in  knowing  what  skills  they 
are  expected  to  perform  and  to  what  standards. 
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Physical  Therapy  Rating  Scales 

A  survey  of  clinical  evaluation  forms  received  from  seven  of  the 
ten  Physical  Therapy  Departments  at  Canadian  universities  revealed 
that  some  type  of  rating  scale  is  used  to  evaluate  students  during  the 
clinical  component  of  the  education  program.  The  common  form  was  a 
rating  scale  with  space  provided  for  comments. 

Each  Physical  Therapy  Department  has  developed  its  own  clinical 
rating  form.  Forms  varied  greatly  in  format,  content  and  degree  of 
objectivity.  Specifically,  the  forms  differed  in  the  competencies 
evaluated  and  only  one  of  the  forms  indicated  the  degree  of  importance 
of  each  competency.  None  of  the  forms  had  as  criteria,  specific 
observable  behavior,  which  would  indicate  attainment  of  a  competency. 
Three  forms  defined  the  levels  of  competence  that  students  are 
required  to  achieve  for  adequate  performance  upon  entry  into  the 
profession.  There  was  no  evidence  provided  that  validity  or 
reliability  studies  have  been  conducted  on  any  of  these  evaluation 
forms . 

A  review  of  the  PT  education  literature  revealed  few  evaluation 
instruments  to  measure  clinical  competence.  None  of  these  reported 
instruments  has  been  subjected  to  the  rigors  of  test  development  and 
validation. 

Kern  and  Mickelson  (1971)  at  Temple  University  described  the 
development  and  use  of  a  combined  checklist-rating  scale  for  the 
primary  purpose  of  diagnosing  the  strengths  and  weaknesses  of  PT 
students.  Contained  within  the  instrument  were  ten  terminal  be¬ 
havioral  objectives  and  their  component  behaviors,  which  were  grouped 
into  four  categories:  professional  practices,  treatment  skills. 
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supervisory  practices  and  specific  techniques.  The  performance 
criteria  for  the  rating  scales  were  based  on  levels  of  supervision 
required  in  order  for  the  student  to  meet  the  objectives.  No  study 
was  done  to  demonstrate  the  reliability  and  validity  of  this 
instrument. 

Mays  (1973)  conducted  a  study  to  determine  the  reliability  of 
the  rating  scale  used  to  evaluate  senior  PT  students  from  Virginia 
Medical  College.  The  rating  scale  consisted  of  eleven  objectives, 
each  with  two  or  more  criteria.  Each  student  was  evaluated  by  two 
different  raters,  one  for  each  six  week  affiliation  in  the  summer  of 
1970.  Forty-three  physical  therapists  completed  evaluations  on  thirty 
students  and  ranked  the  objectives  from  the  most  important  to  the 
least  important  with  none  being  given  the  same  rank.  The  inter-rater 
reliability,  Kendall  coefficient  at  .604,  was  significantly  different 
from  zero  at  the  .001  level.  The  four  objectives  judged  to  be  signif¬ 
icantly  more  important  than  the  other  objectives  were  application  of 
basic  knowledge  to  PT,  application  and  teaching  of  PT  procedures, 
needs  of  the  patient  and  observation  (p  £  .01  by  the  Kolmogorov- 
Smirnov  test).  The  grades  for  the  top  four  ranking  objectives  had 
high  correl ational  values  with  the  overall  grades  received  by  the 
thirty  students  at  Rho=.69  to  .90;  however,  lesser  ranked  objectives 
also  has  high  correl ational  values  at  Rho=.56  to  .91,  which  the  author 
interpreted  as  a  halo  effect.  She  concluded  that  the  evaluation  form 
could  be  shortened  from  eleven  to  four  objectives  to  yield  more 
reliable  results.  No  further  studies  have  been  reported. 

Bemis,  Smith  and  Mauser  (1978)  developed  a  cri terion-referenced 
form  for  clinical  evaluation  of  PT  students  at  Georgia  Medical 
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College.  The  form  consisted  of  twenty-nine  questions  divided  into 
five  competency  sections:  patient  evaluation,  program  planning, 

implementation  of  treatment  program,  interpersonal  relations  and 
professional  behavior.  Under  each  question  there  were  four  descrip¬ 
tions  of  the  possible  behaviors  ranging  from  incompetent  to  highly 
competent.  The  rater  was  asked  to  choose  the  description  which  best 
described  the  student's  performance  as  observed  by  the  rater.  A 
"not-observed"  option  was  available.  A  space  was  provided  at  the  end 
of  each  section  for  comments  that  would  clarify  the  reasoning  for 
choices  made.  The  reliability  and  validity  of  this  instrument  was  not 
reported. 

The  Chartered  Society  for  Physiotherapy  in  Great  Britain 
designed  a  Continuous  Assessment  Record  Book  (C.A.R.B.)  in  1976  which 
is  a  check-out  of  student's  performance  of  specific  treatment  proce¬ 
dures  to  determine  if  a  safe  and  effective  level  of  performance  has 
been  reached.  The  Texas  Consortium  for  Physical  Therapy  Clinical 
Education  modified  the  C.A.R.B.  in  1980,  placing  greater  emphasis  on 
the  continued  assessment  of  the  student's  progress  toward  adequate 
mastery  of  essential  skills  throughout  clinical  experiences.  The 
Mastery  and  Assessment  of  Clinical  Skills  (MACS)  is  more  comprehensive 
than  C.A.R.B.,  in  that  it  includes  skills  related  to  professional 
behavior,  communication  and  interpersonal  relations,  program  planning, 
teaching,  administration,  research  and  special  situations.  Each  of 
the  54  skills  included  in  the  MACS  was  defined  by  key  indicators  of 
entry-level  performance  and,  was  checked  according  to  whether 
satisfactory  entry-level  performance  had  been  met  or  not.  No 
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statistics  were  reported  in  the  manual  accompanying  either  evaluation 
system. 

Scully  and  Shepard  (1983)  found  that  evaluations  are  prevalent 
in  PT  clinical  education.  Verbal  exchange  with  the  student  or  obser¬ 
vation  of  patient-student  interaction  were  mostly  used  to  determine 
the  student's  readiness  to  perform  in  different  clinical  situations. 
Evaluations  were  private  and  in  an  atmosphere  that  permitted  certain 
errors  and  did  not  cause  undue  psychological  stress.  Clinical 
therapists  expressed  a  strong  professional  responsibil ity  to  certify 
competence  in  clinical  education  but  expressed  difficulty  in 
recounting  and  writing  their  impressions  of  the  student's  behavior  and 
progress  on  selected  competencies  in  narrative  evaluations,  mainly 
because  it  infringed  on  patient  care  time  and  personal  time. 

In  contrast  to  the  Physical  Therapy  evaluations,  the  American 
Occupational  Therapy  Association  (AOTA)  has  developed  the  Field  Work 
Performance  Ratings  (FWPR)  which  has  been  subjected  to  statistical 
scrutiny  (Deitz  &  Slaymaker,  1979;  Muthard,  Morris,  Crocker  & 
Slaymaker,  1976;  Slaymaker,  Crocker  &  Muthard,  1974).  Items  from  many 
different  evaluation  forms  were  reduced  to  53  items  and  divided  into 
five  performance  areas:  data  gathering,  treatment  planning,  treatment 
implementation,  communication  skills  and  professional  characterist¬ 
ics.  Each  item  was  a  behavioral  statement  of  an  aspect  of  clinical 
performance  and  was  rated  on  a  4-point  scale  ranging  from  rarely 
correct  performance  to  consistent  performance.  The  rating  scale  was 
tested  on  934  Occupational  Therapy  (OT)  students  between  June  1  and 
December  31,  1972.  The  inter-rater  reliability,  obtained  by 
correlating  independent  ratings  from  two  therapists  who  rated  each 
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student,  ranged  from  .60  for  data  gathering  to  .72  for  both 
communication  skills  and  professional  character!- sties  with  an  overall 
reliability  at  .76.  The  level  of  internal  consistency  of  items  within 
the  five  performance  areas  ranged  from  .87  to  .97  as  calculated  by 
Hoyt's  analysis  of  variance.  Correlation  between  FWPR  scores  and  mean 
hiring  ratings  completed  by  the  two  therapists  who  rated  student 
performance  on  the  FWPR  was  .76,  and  correlation  between  FWPR  and 
total  RPSA  scores  (the  original  scale  adopted  by  AOTA)  was  .80,  thus 
demonstrating  good  concurrent  validity.  Because  scores  on  FWPR 
clustered  toward  the  high  end  of  the  scale,  Dietz  and  Slaymaker  (1979) 
revised  the  rating  to  a  5-point  scale.  Results  from  67  OT  students 
evaluated  by  both  the  original  and  revised  FWPR  forms  indicated  that 
student  scores  were  dispersed  more  effectively  and  discriminated  more 
accurately.  The  revised  form  demonstrated  significantly  greater 
variance;  however,  scores  still  tended  to  cluster  at  the  high  end  of 
the  scale.  Muthard,  Morris,  Crocker  and  Slaymaker  (1976)  reported 
that  the  correlations  between  the  previously  administered  FWPR  and  the 
supervisor's  job  rating  of  clinical  performance  of  208  working 
Occupational  Therapists  revealed  a  lack  of  predictive  validity  at  .09 
but  the  correlation  was  not  significant  at  the  .05  probability  level. 

Conclusion 

Evaluations  of  student  performance  serve  to  assist  students  to 
become  safe  and  competent  practitioners  and  to  assure  society  that 
only  those  students  who  are  competent  will  graduate.  Yet,  none  of  the 
evaluations  in  PT  assess  clinical  competence  required  for  adequate 
performance  upon  entry  into  the  profession  has  been  achieved  by 
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students.  Furthermore,  systematic  studies  have  not  been  conducted  to 
determine  the  reliability  and  validity  of  the  clinical  evaluations 
reported  in  the  PT  literature  and  professional  documents.  An 
evaluation  instrument,  which  determines  if  a  PT  student  can  perform 
entry-level  competencies  in  the  clinical  setting,  should  be  developed 
and  tested.  Observational  evaluation  in  the  naturalistic  setting 
provides  the  only  opportunity  to  see  if  clinical  competencies  can 
actually  be  performed  in  the  complex  clinical  environment.  Rating 
scales  are  suited  to  observation  and  evaluation  in  the  clinical 
setting,  which  is  characterized  by  a  wide  range  of  patients, 
situations  and  observers.  Inconsistency  of  observer  ratings  may  be 
lessened  by  careful  definition  of  behaviors  to  be  evaluated  along  with 
the  standards  of  performance  by  which  to  judge  them. 


Chapter  III 


METHOD 

Instrument  Development  Phase 


To  develop  an  instrument  to  evaluate  clinical  competence  of 
fourth  year  PT  students  at  the  UA,  four  steps  were  followed:  (1)  A 
list  of  competencies  in  PT  was  prepared  by  a  survey  of  the  literature 
and  professional  documents  and  was  verified  by  physical  therapists 
from  diverse  areas  of  clinical  practice  and  education,  (2)  A 
questionnaire  was  sent  to  physical  therapists  working  in  Edmonton  to 
validate  the  list  of  competencies,  (3)  An  evaluation  instrument  was 
planned  and  constructed,  and  (4)  Instructions  to  administer  the 
instrument  were  devised. 

Identification  of  Competencies 

A  list  of  226  process  competencies  was  compiled  which  included  a 
wide  array  of  knowledge,  skills,  judgements  and  attitudes  necessary 
for  safe  and  optimal  care  to  patients  by  PT  students.  The  information 
was  gathered  from  evaluation  instruments  reported  in  the  literature 
(Bemis,  Smith  &  Mauser,  1978;  Kern  &  Mickelson,  1971;  Mays,  1973; 
Slaymaker,  1978;  Wilhelm,  1969),  articles  on  clinical  education  & 
practice  (Chidley  &  Kisner,  1979;  Davis,  Anderson  &  Jagger,  1979;  May, 
1977  and  1977;  McDaniel,  1964)  and  professional  documents  (American 
Physical  Therapy  Association  Competence  Manual,  1981;  American 
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Physical  Therapy  Association  Standards  for  Basic  Education  in 
Physical  Therapy,  1972;  American  Physical  Therapy  Association 
Standards  for  Physical  Therapy  Practitioners,  1972;  Canadian 
Physiotherapy  Association  Recommended  Core  Curriculum  for 
Physiotherapy  Education  Programmes,  1982;  Chartered  Society  for 
Physiotherapy  Continuous  Assessment  Record  Book,  1976). 

This  initial  list  was  sorted  and  condensed  into  a  list  of  86 
competencies  by  eliminating  repetitive  or  similar  competencies 
(Appendix  A).  The  condensed  list  was  analyzed  by  five  physical 
therapists  from  diverse  areas  of  clinical  practice  and  education,  who 
were  asked  to  delete  insignificant  competencies,  to  reword 
competencies  for  clarity  and  meaning  and  to  suggest  any  competencies 
that  may  have  been  overlooked.  In  view  of  their  responses,  the  list 
was  reduced  further  to  a  final  55  competency  statements,  which  were 
then  written  in  terms  of  the  observable  behaviors  that  a  student  must 
display  to  fulfill  each  competency.  This  final  list  of  competencies 
was  categorized  into  nine  major  competency  areas,  which  define  the 
domain  of  clinical  competence:  patient  evaluation,  program  planning, 
implementation  of  the  treatment  program,  communication  with  patient 
and  family,  communication  with  health  care  personnel,  documentation, 
professional  behavior,  professional  growth,  and  management  related  to 
direct  patient  care. 

Questionnaire 

The  questionnaire  listed  the  55  competency  statements  and  the 
nine  major  competency  areas  (Appendix  B).  The  respondents  were  asked 
to  rate  the  statements  and  areas  on  a  5-point  scale  according  to  how 
important  that  they  felt  each  was  to  clinical  practice  in  PT  (non- 
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essential,  minor  importance,  important,  very  important,  essential). 
Two  hundred  questionnaires  were  sent  to  physical  therapists  who  worked 
at  clinical  facilities  in  Edmonton  offering  clinical  placements  to  UA 
students . 

To  determine  the  ratings  of  importance,  the  frequency  of 
responses  was  calculated  for  each  of  the  55  competency  statements  and 
for  each  of  the  nine  major  competency  areas.  Two  calculations  were 
used  as  indicators  of  a  rating  of  "high  importance".  The  mean  impor¬ 
tance  rating  indicated  the  mean  of  the  importance  ratings  given  to 
each  competence,  and  the  percentage  of  importance  was  the  percentage 
of  respondents  who  scored  a  competency  either  "essential"  or  "very 
important". 

The  modal  rating  of  importance  was  used  as  a  weight  for  each 
competency  and  major  competency  area,  so  that  the  relative  importance 
of  these  various  competencies  influenced  the  score  awarded  to  a 
student.  Use  of  weights  made  an  overall  assessment  of  clinical  com¬ 
petence  possible,  such  that,  the  student's  score  on  each  competency 
was  multiplied  by  its  respective  weight  before  summing  the  total  score 
over  all  competencies. 

Construction  of  the  Instrument 

The  two  previous  steps  provided  a  plan,  crucial  for  content 
validity,  to  ensure  that  all  important  competencies  were  included  in 
the  instrument.  The  ECC  was  designed  for  evaluation  of  fourth  year  PT 
students  to  monitor  their  progression  toward  clinical  competence 
required  for  entry  into  the  PT  profession  upon  completion  of  their 
undergraduate  program. 
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The  ECC  was  planned  in  two  sections  (Appendix  C).  Section  1 
contained  the  patient  care  competency  areas:  patient  evaluation, 
program  planning,  implementation  of  the  treatment  program,  communica¬ 
tion  with  patient  and  family,  communication  with  health  care  personnel 
and  documentation.  Section  II  contained  professional  competencies: 
professional  behavior  and  professional  growth.  The  content  of  each 
major  competency  area  was  delineated  by  its  subcompetency  statements. 
There  were  48  subcompetencies  in  all. 

Section  I  was  designed  as  a  behaviorally-anchored,  4-point 
rating  scale  to  determine  a  final  rating  of  clinical  competence,  which 
reflected  habitual  performance  of  patient  care  skills  observed  and 
recorded  daily  throughout  the  placement.  There  were  34  items  in 
Section  I.  Section  II  was  designed  as  a  subjectively-worded  4-point 
rating  scale  to  determine  a  final  rating  of  the  frequency  with  which 
professional  competencies  were  demonstrated  by  the  student  in  the 
final  week  of  the  placement.  There  were  14  items  to  be  rated  in 
Section  II. 

The  next  step  was  to  decide  the  standards  of  performance  by 
which  to  judge  clinical  performance  of  the  subcompetencies  in  both 
Sections  I  and  II  of  the  ECC.  Four  standards  were  used  and  served  as 
the  anchorpoints  on  the  rating  scale  descriptive  of  levels  of  perfor¬ 
mance  ranging  from  incompetent  to  highly  competent. 

Under  each  subcompetency  in  Section  I,  the  four  standards  of 
performance  were  described  in  behavioral  terms  to  depict  what  a 
student  must  do  to  qualify  for  each  standard:  (1)  incompetent,  (2) 
minimally  competent,  (3)  competent,  and  (4)  highly  competent.  The 
four  standards  were  worded  behavioral  ly,  as  one  means  of  striving  for 
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higher  reliability  of  the  instrument.  Incompetent  performance 
standards  were  defined  by  the  need  for  maximal  assistance  or 
correction  by  the  preceptors  to  ensure  an  acceptable  level  of  perfor¬ 
mance;  minimally  competent  standards  were  defined  by  the  need  for 
minimal  assistance  or  correction  by  the  preceptors  to  ensure  an 
acceptable  level  of  performance.  These  low  levels  of  performance  were 
based  on  the  amount  of  supervision  given,  in  that,  preceptors  cannot 
allow  poor  performance  to  persist.  Competent  performance  standards 
were  characterized  by  independent  performance  with  safety,  effective¬ 
ness  and  proficiency  in  relation  to  more  complex  tasks  or  changing 
circumstances  or  problems.  A  "zero"  option  was  also  provided  if 
performance  was  not  observed  by  the  preceptor  or  if  the  student  was 
not  required  to  perform  a  subcompetency. 

The  student's  score  on  each  subcompetency  in  Section  I  was 
judged  by  the  standard  (0,1, 2, 3, 4)  which  best  described  the  student's 
performance  as  observed  by  the  preceptor.  The  format  of  Section  I  was 
a  progressive  format  consisting  of  columns  adjacent  to  the  subcom¬ 
petencies  for  recording  the  student's  daily  scores  and  the  final  score 
at  the  end  of  the  placement.  This  formative  evaluation  told  students 
and  preceptors  alike  what  subcompetencies  were  expected  and  to  what 
standard  of  performance  as  well  as  how  well  the  student  had  attained 
each  subcompetency. 

In  Section  II,  the  four  standards  of  performance  were  subjec¬ 
tively  worded  to  describe  the  frequency  with  which  the  student  showed 
professional  competencies:  (1)  occasionally,  (2)  inconsistently,  (3) 
usually,  and  (4)  always.  Each  standard  was  defined  by  the  number  of 
lapses  in  professional  conduct  that  occurred  in  the  final  week  of  the 
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placement.  The  final  week  was  selected  to  lessen  preceptor  error 
likely  introduced  by  fading  memory  over  a  longer  period  of  time.  A 
student  was  scored  as  competent  if  the  professional  competencies  were 
"usually"  performed.  A  zero  option  was  also  provided,  if  performance 
was  not  observed. 

The  student's  total  score  on  the  ECC  (Section  I  and  II)  was 
obtained  by  summing  the  student's  weighted  score  on  each  subcom¬ 
petency,  hence  providing  a  summative  evaluation  of  the  student's 
overall  level  of  performance. 

In  summary.  Evaluation  of  Clinical  Competence  was  constructed  in 
two  sections.  Section  I  was  a  4-point,  behavioral ly-anchored  rating 
scale  to  determine  the  extent  to  which  a  PT  student  performed  entry- 
level  competencies  as  judged  by  specific  standards  of  performance. 
This  evaluation  involved  a  daily  record  of  performance  to  monitor 
habitual  performance  for  the  determination  of  a  final  rating.  Section 
II  was  a  4-point,  subjectively-worded  rating  scale  to  determine  a 
final  rating  of  the  professional  competencies  displayed  by  the  student. 

Instructions  to  Administer  the  Instrument 

The  final  stage  in  the  instrument  development  phase  was  the 
formulation  of  instructions  to  administer  the  ECC.  Precise  instruc¬ 
tions  were  documented  in  a  directive  to  perceptors  to  ensure  that 
administration  conditions  were  uniform  for  all  students  (Appendix  D). 
This  step  was  done  as  a  further  attempt  to  improve  the  reliability  of 
the  instrument. 

Section  I  and  Section  II  of  the  ECC  had  separate  instructions. 
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Instructions  for  Daily  Ratings:  Section  I 

Two  preceptors  observed  and  rated  each  student  once  daily  from 
day  4  to  day  17  of  a  placement,  consisting  of  20  days.  A  total  of  14 
daily  ratings  of  student  performance  was  possible  with  different 
patients  and  in  different  competency  areas. 

The  two  preceptors  observed  the  student's  performance  for 
fifteen  minutes  each  day.  The  first  preceptor  scheduled  the  daily 
observations  at  a  time  agreeable  to  the  preceptors  and  the  student. 
The  onus  was  on  all  parties  to  be  prepared  for  observation  at  the 
scheduled  time.  Preceptors  observed  the  student  performing  in  any  one 
of  the  following  activities:  (1)  patient  evaluation,  (2)  patient 
treatment,,  (3)  discussion  of  program  planning,  (4)  team  conferences  or 
team  interactions,  and  (5)  reading  student's  documentation  in  patient 
charts.  These  methods  of  observation  were  varied  from  day  to  day  by 
the  first  perceptor,  so  that  each  major  competency  area  was  observed  a 
minimum  of  two  times  at  different  intervals  throughout  the  placement. 
The  selection  of  patients  was  also  varied  from  day  to  day  by  the  first 
preceptor,  so  that  the  student  was  observed  with  different  types  of 
patients  over  the  placement.  The  student  was  free  to  refuse  being 
observed  with  any  patient  that  he  or  she  did  not  consider  appropriate. 

The  preceptors  were  instructed  to  observe  the  student  together; 
however,  if  it  was  not  feasible  due  to  constraints  of  specialty  units, 
such  as  intensive  care,  preceptors  were  allowed  to  make  separate 
fifteen  minute  appointments  to  observe  the  student  individually.  In 
this  way,  daily  observation  and  ratings  could  still  be  made.  Guide¬ 
lines  documented  ways  to  lessen  the  student's  anxiety  during  these 
schedule  observation  times. 
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Immediately  or  as  soon  as  possible  after  observing  the  student's 
performance,  the  two  preceptors  independently  rated  the  student's 
performance  on  all  relevant  subcompetencies  in  Section  I  of  the  ECC. 
No  joint  discussion  or  joint  decisions  were  made  by  the  two  preceptors 
during  either  the  observation  or  rating  of  the  student's  performance. 
Each  preceptor  independently  chose  the  standard  of  performance 
(0,1 ,2,3,4).  from  the  choices  provided  under  each  subcompetency  which 
best  described  the  student's  performance  on  the  subcompetency.  When 
choosing  a  standard,  the  preceptors  were  asked  to  base  his  or  her 
decision  on  the  definition  of  terms  provided  on  page  one  of  the 
instrument.  Adjacent  to  each  subcompetency,  the  preceptor  wrote  the 
number  representing  the  standard  of  performance  that  was  chosen.  This 
number  was  written  in  the  column  correspondi ng  to  the  day  of  the 
evaluation.  If  observations  by  preceptors  were  conducted  separately, 
this  condition  was  indicated  by  circling  the  "s"  (separate)  at  the  top 
of  the  same  column  as  the  daily  rating. 

Instructions  for  Final  Ratings:  Section  1 

Final  ratings  were  completed  anytime  between  day  18  and  20.  The 
preceptors  independently  reviewed  the  student's  daily  ratings  and  then 
independently  chose  a  rating  which  best  described  the  student's 
overall  performance  in  each  subcompetency.  This  number  was  written  in 
the  column  corresponding  to  day  20. 

Instructions  for  Final  Ratings:  Section  II 

Final  ratings  were  completed  along  with  Section  I  of  the  ECC 
anytime  between  day  18  and  day  20.  The  preceptors  independently 
considered  only  the  final  week  of  the  placement  to  rate  the  frequency 
with  which  professional  subcompetencies  were  demonstrated  by  the 
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student  and  placed  a  check  mark  in  the  appropriate  column  (not 
observed,  occasionally,  inconsistently,  usually  and  always). 


Instrument  Validation  Phase 


Informed  Consent 

The  supervisors  of  all  the  physical  therapy  departments  in 
Edmonton  providing  clinical  placements  to  PT  students  from  the  UA  were 
contacted  by  a  letter  informing  them  of  the  study  and  requesting  their 
participation.  All  consented  to  participate  in  the  study.  The  six 
departments  outside  Edmonton  offering  placements  were  also  contacted, 
and  consent  was  obtained  from  four  of  them.  The  supervisors  from 
these  four  physical  therapy  departments  outside  Edmonton  obtained 
consent  from  the  two  therapists  in  their  respective  departments  who 
would  be  supervising  the  student.  A  followup  visit  to  each  department 
in  Edmonton  agreeing  to  participate  in  the  study  was  made  in  August 
1982  to  explain  the  evaluation  instrument  and  its  instructions  to  the 
physical  therapists  responsible  for  supervision  of  the  students. 
Consent  was  solicited  and  obtained  from  all  the  therapists  that  would 
be  involved. 

In  September  1982,  the  PT  students  were  informed  as  to  the 
nature  of  the  study  and  their  consent  to  participate  was  obtained. 
Thirty-two  fourth  year  students  (n=35)  agreed  to  participate;  23  third 
year  students  (n_=35)  agreed  to  participate.  Students  were  told  that 
the  information  gathered  by  the  ECC  would  not  be  used  to  determine  a 
grade  in  their  practicum  courses;  the  existing  evaluation  form  would 
be  completed  concurrently  and  utilized  for  grading  purposes. 
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Moreover,  they  were  informed  that  they  could  withdraw  from  parti¬ 
cipation  at  any  time  during  the  study.  Results  from  the  revised 
evaluation  would  be  mailed  to  students,  if  so  requested  (Appendix  E). 

Pilot  Testing  of  the  Evaluation  of  Clinical  Competence 

In  September  1982,  the  ECC  was  tested  on  four  fourth  year  PT 
students  at  the  UA  Hospital  to  ascertain  the  feasibility  of  the 
instrument,  administration  procedures  and  instructions  prior  to  the 
validation  studies.  Some  ambiguous  subcompetencies  and  standards  were 
identified  and  revised.  Instructions  to  preceptors  were  modified  to 
allow  the  separate  observations  by  preceptors  in  special  instances, 
e.g.  intensive  care,  and  to  include  the  suggestions  on  how  to  ease  the 
student's  anxiety  during  the  scheduled  observations  by  two  preceptors. 

Validation  Studies  of  the  Evaluation  of  Clinical  Competence 

To  validate  this  newly-designed  instrument,  three  steps  were 
followed.  First,  the  administration  procedures  for  the  validation 
studies  were  established.  Second,  the  ECC  was  administered  to  fourth 
year  PT  students  (n_=29)  during  their  clinical  placement  in  November 
1982  according  to  the  specified  instructions  to  obtain  data  by  which 
to  analyze  and  revise  the  evaluation  instrument.  Third,  the  revised 
ECC  was  administered  to  third  year  PT  students  (n=23)  and  fourth  year 
PT  students  (n=25)  during  their  clinical  placements  in  February  and 
March  1983.  The  same  instructions  were  followed  in  order  to  collect 
data  necessary  to  establish  inter-rater  reliability,  concurrent 
validity  and  construct  validity  of  the  instrument. 
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Administration  Procedures 

Third  and  fourth  year  PT  students  were  observed  and  rated  during 
each  clinical  placement  by  two  preceptors  who  were  assigned  to  the 
students  by  the  Clinical  Instructor  in  each  clinical  facility.  The 
first  preceptor  was  the  therapist  normally  assigned  to  supervise  the 
student  throughout  the  placement.  The  second  preceptor  was  a  clinical 
instructor,  team  supervisor  or  another  staff  therapist  working  in 
close  proximity  and  in  the  same  specialty  area  who  was  not  assigned 
another  student  in  the  same  year  to  supervise  directly. 

Both  preceptors  evaluated  the  third  and  fourth  year  PT  students 
using  two  evaluations:  (1)  the  Evaluation  of  Clinical  Competence,  and 
(2)  the  Hiring  Rating.  The  Hiring  Rating  indicated  the  preceptor's 
willingness  to  employ  the  student,  as  a  measure  of  concurrent  validity 
(Appendix  F).  Further  to  these  two  evaluations,  a  narrative  clinical 
evaluation  used  by  the  Department  of  Physical  Therapy  at  the  UA,  was 
completed  concurrently  to  grade  the  student's  clinical  experience  for 
the  practicum  courses  (Appendix  G).  This  additional  evaluation  was 
completed  by  the  first  preceptor  only. 

At  the  end  of  the  placement,  the  first  and  second  preceptors 
independently  determined  final  ratings  on  their  respective  evalua¬ 
tions,  in  accordance  with  a  preassigned  counterbalanced  sequence  so  as 
to  avoid  any  sequence  bias  (Appendix  H).  Furthermore,  to  ensure  that 
information  from  one  evaluation  did  not  unjustifiably  influence  the 
information  on  another  evaluation,  preceptors  were  told  to  completely 
finish  one  evaluation  before  proceeding  to  the  next.  Preceptors  were 
warned  not  to  refer  to  an  earlier  evaluation  nor  to  change  an  evalua¬ 
tion  once  completed.  The  instructions  by  which  to  complete  the  three 
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evaluations  were  clearly  documented  (Appendix  D)  and  were  explained  to 
the  preceptors  by  the  investigator  in  a  one  hour  training  session 
prior  to  the  commencement  of  each  placement  in  the  validation  study. 

Evaluation  and  Revision  of  the  Instrument 

The  ECC  was  administered  to  32  fourth  year  PT  students  during 
their  twelfth  clinical  placement  in  November  1982.  The  results  for 
three  students  were  excluded,  because  two  preceptors  could  not  be 
proviaed.  The  data  collected  were  analyzed  to  evaluate  how  consis¬ 
tently  the  first  edition  of  the  instrument  measured  clinical 
competence;  thereafter,  revisions  were  made  to  improve  its  performance. 

To  identify  inconsistent  subcompetencies  (items)  the  final 
ratings  awarded  to  each  student  by  the  two  preceptors  were  analyzed. 
Crosstabulation  tables  were  drawn  for  each  item  using  the  Statistical 
Package  for  Social  Sciences  (SPSS).  For  each  item,  the  percentage 
agreement  between  the  two  preceptors  was  calculated,  and  the 
discrepant  ratings  responsible  for  disagreement  were  located.  Incon¬ 
sistent  items  were  deleted,  and  those  retained  were  reworded  or 
refined  in  light  of  the  crosstabulation  results. 

Validation  of  the  Revised  Instrument 

The  revised  ECC  (Appendix  I)  was  administered  under  the  same 
administrative  conditions  on  two  occasions  in  February  and  March, 
1983:  (1  )  with  23  third  year  PT  students  in  their  fifth  clinical 
placement,  and  (2)  with  30  fourth  year  PT  students  in  their  fifteenth 
and  final  clinical  placement.  In  the  fourth  year  group,  results  for 
five  students  were  excluded,  because  two  preceptors  could  not  be 
provided  in  certain  clinical  facilities.  The  data  collected  were 
analyzed  to  determine  the  inter-rater  reliability  and  validity  of  the 


■ 


, 


48 


revised  instrument  with  these  two  groups  of  students. 

To  establish  inter-rater  reliability  of  the  ECC,  the  final 
ratings  awarded  to  each  student  by  the  two  preceptors  were  analyzed. 
Again,  crosstabulation  tables  were  used  to  calculate  the  percentage  of 
agreement  between  preceptors  and  to  locate  rating  cells  with  a  high 
percentage  of  disagreement.  Kappa  coefficients  were  also  computed 
from  the  crosstabulation  tables. 

Cohen  (1960)  presented  a  Kappa  coefficient  to  measure  the  degree 
of  agreement  between  two  judges  for  nominal  scales.  He  argued  that 

O 

use  of  X  is  inappropriate  for  evaluation  of  agreement.  A  signi- 
ficant  X  signifies  departure  from  chance  association,  be  it 
disagreement  or  agreement.  A  Kappa  coefficient  is  the  proportion  of 
agreement  after  chance  agreement  is  removed  from  consideration: 

k  =  (po  -  pc) / ( 1  -  pc)  where 

po  equals  the  proportion  of  units  in  which  judges  agree;  and 

pc  equals  the  proportion  of  units  for  which  agreement  is 
expected  by  chance 

The  daily  ratings  per  se  were  not  used  in  the  analysis  of  the 
data.  Information  regarding  the  student's  competence  was  accumulated 
daily  to  be  used  by  the  preceptors  for  determining  the  final  rating 
indicative  of  the  student's  habitual  performance  throughout  the 
clinical  placement. 

The  students'  final  raw  scores  were  transformed  using  the  modal 
weights  to  produce  for  each  student  a  total  scale  score  and  seven 
subscale  scores  for  each  of  the  major  competency  areas.  Intraclass 
correl ations ,  derived  by  one-way  analysis  of  variance  with  repeated 
measures,  were  calculated  to  determine  the  reliability  of  the  total 
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scale  score  and  seven  subscale  scores  awarded  by  the  two  preceptors. 

To  determine  concurrent  validity  of  the  ECC,  it  was  predicted 
that  students  who  rated  highly  on  the  ECC  would  also  tend  to  rate 
highly  on  the  criterion  variables  of  potential  empl oyabi 1 ity  and  of 
student  performance  by  a  narrative  evaluation  method.  Concurrent 
validity  coefficients  were  Pearson  product-moment  correlations  between 
the  total  ECC  scores  and  the  hiring  scores  and  correlations  between 
the  total  ECC  scores  and  the  score  on  the  narrative  evaluation. 

To  ascertain  whether  the  ECC  actually  measured  the  underlying 
construct  of  clinical  competence,  construct  validity  was  tested  by 
predicting  that  experienced  students  perform  better  than  novice 
students.  One-way  analysis  of  variance  was  used  to  determine  if  there 
was  a  significant  difference  between  the  final  scale  score  received  by 
the  experienced  fourth  year  students  and  the  final  scale  scores 
received  by  the  novice  third  year  students. 

Immediately  following  completion  of  the  validation  study,  both 
preceptors  and  students  were  asked  to  complete  a  short  questionnaire 
about  their  feelings  toward  the  ECC,  its  advantages  and  disadvantages 
(Appendix  J).  The  frequency  of  responses  was  calculated  for  students 
and  preceptors  separately  to  gauge  their  acceptance  of  the  ECC. 
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Chapter  IV 


RESULTS 

Instrument  Development  Phase 

The  total  number  of  replies  to  the  questionnaire  regarding  the 
importance  of  clinical  competencies  in  physical  therapy  was  121  or 
60.5%. 

The  importance  placed  on  each  of  the  nine  major  competency  areas 
is  demonstrated  in  Table  1  by  their  ranking  according  to  the  mean 
importance  rating.  The  four  highest  ranking  clinical  competencies, 
namely  patient  evaluation,  implementation  of  treatment,  program  plan¬ 
ning  and  communication  with  patient/family  had  mean  importance  ratings 
between  4.79  and  4.08,  out  of  a  possible  5.00.  The  five  lowest  rank¬ 
ing  clinical  competency  areas  included  documentation,  communication 
with  health  care  personnel,  professional  behavior,  professional  growth 
and  mangement  related  to  direct  patient  care  with  mean  importance 
ratings  ranging  from  3.97  to  3.54,  out  of  a  possible  5.00. 

The  importance  of  the  subcompetencies  under  each  of  the  major 
competency  areas  portrays  the  priorities  espoused  by  members  of  the  PT 
profession.  Each  major  competency  area  is  considered  in  turn. 

The  patient  evaluation  area,  consisting  of  nine  subcompetencies 
or  items  related  to  initial  patient  evaluation  and  re-evaluation 
procedures,  was  ranked  the  highest  in  importance  with  mean  importance 
ratings  ranging  from  3.78  to  4.55.  More  than  68%  of  the  respondents 
scored  eight  items  highly  important;  however,  only  63%  considered 
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Table  1 

Importance  Ratings  of  Physical  Therapy  Competencies 


Major  Competency  Areas 

Ranked  from  High  to  Low 

No.  of 
Items 

Competency 

Means 

Range  of 

Subcompetency  Means  of  % 
Means  Importance 

patient  evaluation 

9 

4.79 

3.78 

-  4.55 

77.9 

implementation  of  treatment 

9 

4.60 

3.26 

-  4.65 

74.4 

program  planning 

8 

4.53 

3.65 

-  4.56 

80.4 

communication  with  patient 
and  family 

6 

4.08 

3.71 

-  4.16 

72.3 

documentation 

3 

3.97 

3.78 

-  4.22 

71.6 

conmuni cation  with  other 
health  care  personnel 

5 

3.94 

3.51 

-  4.57 

72.7 

professional  behavior 

6 

3.92 

3.79 

-  4.28 

70.5 

professional  growth 

5 

3.85 

3.44 

-  4.27 

67.6 

management  related  to 
direct  patient  care 

4 

3.54 

3.34 

-  4.03 

55.2 

Note:  n=l 2 1 
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explanation  of  evaluation  procedures  and  findings  to  the  patient/ 
family  to  be  highly  important.  The  most  highly  valued  items  involved 
selection  of  PT  evaluation  procedures  and  effective  performance  of  an 
evaluation  at  93%  and  92%,  respectively. 

The  program  planning  area,  with  its  eight  items  pertaining  to 
planning  treatment  services,  was  ranked  third  in  importance  with  mean 
importance  ratings  between  3.65  and  4.56.  Although  six  items  were 
rated  highly  important  by  80%  of  the  respondents,  only  55%  considered 
inclusion  of  the  patient/family  in  some  of  the  planning  process  to  be 
highly  important.  By  contrast,  the  most  highly  valued  item  was 
identification  of  primary  problems  in  priority  at  93%. 

The  implementation  of  treatment  area  included  nine  items, 
related  to  treatment  skills  and  judgements,  and  ranked  second  in 
importance  with  mean  importance  ratings  ranging  from  3.26  to  4.65. 
Only  two  items  did  not  receive  highly  important  ratings  by  65%  of  the 
respondents:  preparation,  care  and  tidiness  of  equipment  and  treat¬ 
ment  area  at  41%  and  organization  of  patient  schedule  at  52%.  The 
most  highly  valued  items  were  effective  performance  of  a  treatment  and 
adherence  to  all  safety  precautions  both  at  93%. 

The  communication  with  patient/family  area,  consisting  of  six 
items  related  to  communication  skills  and  interpersonal  relations, 
ranked  fourth  in  importance  with  mean  importance  ratings  between  3.71 
and  4.16.  All  items  were  viewed  as  highly  important  by  71%  to  79%  of 
the  respondents,  except  one  item  which  pertains  to  education  of  the 
patient/family  regarding  patient's  status  and  PT  treatment,  was  viewed 
as  highly  important  by  61%. 

The  communication  with  health  care  personnel  area,  with  its  five 
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items,  ranked  sixth  in  importance  with  mean  importance  ratings  ranging 
from  3.51  to  4.57.  The  most  highly  valued  item  was  confidentiality  of 
patient  information  at  87%  and  the  least  valued  item  was  assisting 
other  personnel  implement  aspects  of  the  patient's  PT  treatment  at  53%. 

The  documentation  area  was  comprised  of  three  items  and  ranked 
fifth  with  mean  importance  ratings  from  3.78  to  4.22.  The  items 
referring  to  content  of  the  initial  evaluation  and  progress  notes 
received  79%  and  78%  highly  important  rating  respectively;  whereas, 
documentation  of  statistical  and  patient  records  according  to  the 
format  ana  time  limits  of  the  facility  received  only  a  58%  highly 
important  rating. 

The  professional  behavior  area  included  six  items  covering 
professional  qualities  that  enhance  working  relations.  It  ranked 
seventh  in  importance  with  mean  importance  ratings  ranging  from  3.79 
to  4.28.  High  ratings  were  given  to  items  related  to  accepting 
responsibility  at  85%  and  assisting  and  cooperating  with  co-workers  at 
76%.  Low  ratings  of  61%  were  given  to  both  presenting  a  professional 
appearance  and  handling  frustrations  appropri ately . 

The  professional  growth  area  involved  five  items  characteri stic 
of  qualities  that  promote  continuing  clinical  competence.  It  ranked 
eighth  in  importance  with  mean  importance  ratings  ranging  from  3.44  to 
4.27.  Only  the  item  on  interpretation  and  utilization  of  research 
results  was  the  rating  of  importance  less  than  62%  of  respondents. 

Management  related  to  direct  patient  care,  composed  of  four 
items  dealing  with  administrative  roles  for  the  provision  of  patient 
care,  ranked  the  least  important  with  mean  importance  ratings  between 
3.34  and  4.03.  The  highly  rated  item  was  collaboration  with  other 
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disciplines  involved  in  the  patient  care  team  at  74%.  The  low  items 
were  supervision  of  support  PT  personnel  at  41%  and  adherence  to 
operational  procedures  at  45%. 

In  light  of  the  questionnaire  results,  the  list  of  55  subcom¬ 
petencies  was  reduced  to  48  subcompetencies  for  inclusion  in  the 
evaluation  instrument.  The  management  competency  area  was  deleted, 
and  its  one  important  item  on  collaboration  with  other  disciplines  was 
incorporated  into  communication  with  health  care  personnel. 

The  weights  attached  to  each  subcompetency  were  the  modal  rating 
of  importance  received  by  each  subcompetency  in  the  questionnaire. 
These  weights,  which  reflect  the  relative  importance  of  the  subcom¬ 
petency,  are  presented  as  part  of  Table  3,  which  will  be  described  in 
more  detail  in  the  next  section. 

Instrument  Validation  Phase 

Evaluation  of  the  Instrument 

To  assess  how  consistently  the  new  instrument  measured  clinical 
competence  of  29  fourth  year  PT  students  in  November  1982,  the  ratings 
were  analyzed  to  determine  which  subcompetencies  had  high  agreement 
among  preceptors  and  which  had  low  agreement  among  preceptors. 

In  the  analysis,  the  percentage  agreement  was  calculated  when 
the  "not  observed"  rating  was  removed  from  the  data  to  give  an 
inaication  of  agreement  when  a  subcompetency  was  actually  observed  and 
rated  by  the  preceptors.  A  pattern  of  agreement  of  the  48 
subcompetencies  or  items  is  demonstrated  by  their  ranking  according  to 
the  percentage  of  agreement  results.  Table  2  indicates  for  each  item 
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Item 

19 

4 

24 

22 

37 

23 

28 

1 

6 

42 

2 

26 

30 

36 

8 

5 

27 

3 

9 

16 

17 

20 

14 

15 

18 

44 

46 

39 

25 

29 
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Table  2 


Subcompetencies  Ranked  by  Percentage  Agreement  Among 
Preceptors  on  the  Evaluation  of  Clinical  Competence 
with  Fourth  Year  PT  Students 


Frequency  of 

% 

%  Disagreement 

Description 

Not  Observed 

Agreement 

and  Cell 

tidies  treatment  area 

8 

90.5 

F 

9.5 

selects  evaluation  procedures 

0 

75.9 

F 

20.7 

educates  patients 

5 

75.0 

F 

20.8 

uses  body  mechanics 

3 

73.1a 

F 

19.2 

presents  professional  appearance 

0 

72.4 

F 

27.6 

prepares  for  discharge 

22 

71.4 

D/F 

14.3 

develops  rapport  with  patient 

8 

71.4 

F 

23.8 

uses  knowledge  for  evaluation 

4 

68.0 

F 

20.0 

evaluates  patient's  equipment 

23 

67.3a 

F 

33.3 

asks  questions 

3 

65.4 

F 

19.2 

gathers  data  from  chart 

9 

65.0 

F 

15.0 

instructs  patient/family 

1 

64.3 

F 

14.3 

communicates  orally 

5 

62.5 

F 

33.3 

maintains  confidentiality 

5 

62.5 

F 

33.3 

re-eval uates 

19 

60.0a 

D/F 

20.0 

evaluates  skillfully 

0 

58.6 

F 

27.7 

motivates  patient 

3 

57.7 

F 

23.1 

conducts  patient  interview 

8 

57.1 

F 

23.8 

uses  knowledge  for  planning 

2 

55.6 

D 

22.2 

treats  skillfully 

0 

55.2 

F 

34.5 

observes  ski  1 Iful ly 

0 

55.2 

F 

24.1 

schedules  patient  load 

9 

55.0 

F 

25.0 

justifies  treatment  plan 

3 

53.8 

D 

26.9 

involves  patient  in  planning 

16 

53.8a 

D 

23.1 

modifies/progresses  treatment 
identifies  own  strengths  and 

3 

53.8 

D 

23.1 

weaknesses 

seeks  resources  to  increase 

16 

53.8 

F 

23.1 

knowledge 

16 

53.8 

D 

38.5 

documents  to  format 

3 

53.8a 

F 

11.5 

approaches  to  patient 
assists  patient  implement  own 

1 

53.6a 

F 

28.6 

treatment 

12 

52.9 

D 

29.4 

, 
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Table  2  (continued) 


Item 

Frequency  of 
Description  Not  Observed 

% 

Agreement 

%  Disagreement 
and  Cell 

34 

records  progress  notes 

4 

52.0 

F 

24.0 

40 

shows  reliability 

2 

51.9 

F 

29.6 

10 

identifies  primary  problem 

0 

51.7 

F 

31.0 

7 

eases  patient  apprehension 

3 

50.0 

D 

23.1 

12 

determines  treatment  goals 

3 

50.0 

F 

23.1 

31 

develops  rapport  with  team 

11 

50.0 

F 

33.3 

43 

modifies  behavior  to  meet 
suggestions 

1 

50.0 

F 

35.7 

45 

use  self-evaluation 

17 

50.0 

F 

25.0 

35 

accepts  responsibility 

3 

50.0a 

F 

30.8 

41 

handles  frustration  appropriately  7 

50.0a 

F 

36.4 

48 

displays  interests 

1 

46.4 

F 

32.1 

11 

identifies  secondary  problems 

3 

46.2 

D 

26.9 

38 

assumes  appropriate  PT  roles 

2 

44.4 

F 

40.7 

33 

records  initial  evaluation 

4 

40.0 

F 

28.0 

13 

selects  treatment  procedures 

1 

35.7 

D 

39.3 

21 

adheres  to  safety 

1 

28.6 

F 

42.9 

32 

assists  others  implement 
patient  treatment 

25 

25.0 

E 

50.0 

47 

utilizes  research  results 

24 

20.0a 

D 

80.0 

Note:  n=29 


a  items  deleted  in  revised  edition  of  the  instrument 
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the  number  of  times  the,  "Not  Observed",  rating  was  used,  the 
percentage  of  agreement,  and  discrepant  ratings  pairs  contributing 
most  to  percentage  disagreement  among  preceptors. 

The  percentage  agreement  of  the  48  items  in  the  ECC  ranged  from 
90.5%  to  20%.  The  mean  percentage  agreement  of  all  items  in  the 
instrument  was  54.3%  and  the  median  value  was  53.8%.  The  not  observed 
rating  was  used  more  than  half  of  the  time  in  nine  items:  22,  6,  8, 
15,  44,  46,  45,  32  and  47. 

Cells  in  the  crosstabulation  tables  were  also  scrutinized  to 
determine  which  discrepant  ratings  pairs  contributed  to  the  percentage 
disagreement  among  the  preceptors.  Findings  confirm  the  prediction 
that  the  percentage  of  disagreement  was  low  in  cells  spanning  crucial 
ratings,  namely  between  incompetence  (1)  and  minimally  competent 
ratings  (2)  (cell  A),  between  incompetent  (1)  and  competent  ratings 

(3)  (cell  B),  between  incompetent  (1)  and  highly  competent  ratings  (4) 
(cell  C),  between  minimally  competent  (2)  and  competent  ratings  (3) 

(cell  D)  and  between  minimally  competent  (2)  and  highly  competent 
ratings  (4)  (cell  E).  By  contrast,  the  percentage  disagreement  was 
predicted  to  be  high  between  competent  (3)  and  highly  competent 

ratings  (4)  (cell  F)  which  do  not  span  crucial  ratings.  Table  2 

illustrates  that  cell  F,  in  fact,  contributed  largely  to  the 

percentage  disagreement. 

When  discrepant  rating  pairs  exist,  the  first  preceptors  tended 
to  give  higher  ratings  than  the  second  preceptors  on  28  items.  On 
seven  items,  there  was  no  tendency  to  give  higher  or  lower  ratings  by 
either  the  first  or  second  preceptors.  On  the  remaining  13  items,  the 
first  preceptors  tended  to  give  lower  ratings  than  the  second 
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preceptors.  Interesting,  six  of  the  latter  items  were  related  to  the 
professional  subcompetencies. 

Since  the  main  purpose  of  the  ECC  was  directed  toward  discri¬ 
minating  between  incompetence  and  competent  performances,  disagreement 
between  competent  and  highly  competent  ratings  (cell  F)  is  not  con¬ 

sidered  a  serious  discrepancy.  Consequently,  competent  (3)  and  highly 
competent  (4)  ratings  which  are  both  representative  of  competence, 

albeit  one  being  more  commendable  than  the  other,  were  treated  as  one 
rating  indicative  of  competence.  Table  3  presents  the  percentage  of 
agreement  for  the  48  subcompetencies  by  major  competency  areas  when  3 
and  4  ratings  were  collapsed  into  one  rating.  In  addition,  percentage 
agreement  for  the  major  competency  areas  is  shown. 

The  resultant  higher  percentages  of  agreement  illustrate  gain 
made  when  a  4-point  rating  scale  is  reduced  to  a  3-point  rating 

scale.  The  mean  percentage  agreement  for  the  entire  instrument  was 

78.4%  with  percentage  agreement  for  the  items  ranging  from  100%  to  20%. 

The  students'  raw  scores  were  transformed  using  the  weights 
specified  in  Table  3.  Each  student  received  a  total  scale  score  and 
eight  subscale  scores  for  each  major  competency  area.  Table  4 

presents  the  intraclass  coefficients  for  the  total  score  and  the  eight 
subscale  scores  along  with  their  F  scores  and  probability  levels. 

The  mean  of  the  students'  total  scale  scores  was  3.13  for 

preceptor  1  and  3.05  for  preceptor  2,  out  of  a  possible  4.00  [F  (1,28) 

=  1.56  p  =  .22].  The  total  scale  scores  ranged  between  2.2  and  3.9. 

Only  the  subscale  scores  for  the  patient  evaluation  area  showed  a 
significant  difference  between  the  two  preceptors,  indicative  of  poor 
agreement.  Reliabilities  in  Table  4  show  moderate  agreement  among 
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Table  3 


Percentage  Agreement  Among  Preceptors  on  a  3-point  Scale 
for  the  Evaluation  of  Clinical  Competence  with 
Fourth  Year  PT  Students 


Item 

Weight 

%  Agreement 
(3-point  scale) 

Item 

%  Agreement 

Weight  (3-point  scale) 

Patient 

Eval uation 

Implementation  Treatment 

1 

3 

88.0 

16 

3 

89.7 

2 

2 

80.0 

17 

3 

79.3 

3 

3 

81.0 

18 

3 

65.4 

4 

3 

96.6 

19 

1 

100.0 

5 

3 

86.2 

20 

1 

80.0 

6a 

2 

100.0 

21 

3 

71.4 

7 

2 

69.2 

22a 

2 

92.3 

8a 

3 

80.0 

23 

2 

85.7 

X  =85.1 

X 

=  83.0 

Program  Planning 

Communication  with  Patient/Fami 

9 

3 

74.1 

24 

2 

95.8 

10 

3 

82.8 

25a 

3 

78.6 

11 

2 

57.7 

26 

2 

80.8 

12 

2 

73.1 

27 

2 

80.8 

13 

3 

46.4 

28 

2 

95.2 

14 

2 

69.2 

29 

2 

70.6 

15a 

1 

69.2 

X  =  67.5 

X 

=  83.6 

. 
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Table  3  (continued) 


Item 

%  Agreement 

Weight  (3-point  scale) 

Item 

Wei ght 

%  Agreement 
(3-point  scale) 

Communcation  with  Health 
Professional 

i  Care 

Documentation 

30 

2 

95.8 

33 

3 

68.0 

31 

2 

83.3 

34 

3 

76.0 

32 

2 

25.0 

X  = 

68.0 

X  =  72.0 

Professional  Behavior 

Professional  Growth 

35a 

3 

80.8 

42 

3 

84.6 

36 

3 

95.8 

43 

2 

85.7 

37 

2 

100.0 

44 

2 

76.9 

38 

2 

85.2 

45 

2 

75.0 

39a 

1 

92.3 

46 

2 

53.8 

40 

2 

81 .5 

47a 

1 

20.0 

41  a 

2 

86.4 

48 

3 

78.6 

X  =  88.9 

X  =  67.8 

Note:  n_=29 

a  items  deleted  in  revised  edition  of  the  instrument 
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Table  4 


Intraclass  Coefficients  and  Results  of  One-Way  Analysis  of  Variance 
with  Repeated  Measures  for  the  Total  Scale  Score  and  Subscale 
Scores  on  the  Evaluation  of  Clinical  Competence 
with  Fourth  Year  PT  Students 


Fourth  Year 

Scores 

Intraclass  Coefficentsa 

F 

P 

patient  evaluation 

.628 

4.27 

.05 

program  planning 

.610 

.04 

.84 

implementation  of  treatment 

.554 

1.34 

.26 

communication  patient/family 

.401 

2.88 

.10 

communi cation  health  care  personnel 

.778 

.93 

.34 

documentation 

.723 

1.62 

.21 

professional  behavior 

.291 

.22 

.64 

professional  growth 

.660 

.01 

.91 

total  scale  score 

.704 

1.56 

.22 

Note:  n=29 


a  adjusted  reliability  values  used 
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preceptors. 

Nine  items  were  deleted  from  the  first  edition  of  the  ECC. 
Items  6,  8,  15  and  47  were  dropped  because  they  could  not  be  observed 
on  a  regular  basis  in  the  clinical  setting.  Items  22,  39  and  41  were 
deleted  because  their  omission  did  not  jeopardize  content  validity,  in 
that,  questionnaire  results  did  not  reveal  them  to  be  highly  impor¬ 
tant.  Items  25  and  35  were  deleted  because  the  percentage  agreement 
was  low,  likely  due  to  the  fact  that  specific  behavior  exemplifying 
these  subcompetencies  was  not  stated.  Although  some  of  the  remaining 
items  had  low  percentage  agreement,  they  were  retained  to  secure 
content  validity.  For  example,  items  33,  13  and  21  were  highly 

important  subcompetencies,  hence  crucial  behaviors  for  measuring  the 
domain  of  clinical  competence  in  PT.  Items  with  low  percentage  agree¬ 
ment  were  carefully  reworded  in  order  to  clarify  the  description  of 
the  behaviors  to  be  exhibited  by  students,  particularly  between  those 
standards  of  behavior  where  disagreement  was  prevalent.  The  better 
items  were  refined  in  light  of  the  suggestions  offered  by  both  the 
preceptors  and  students  as  well  as  in  light  of  disagreement  results. 

Moreover,  the  ECC  was  streamlined  to  seven  major  competency 
areas  by  consolidating  the  two  professional  competency  areas  in 
Section  II  into  one  major  area  called  "professional  behavior".  One 
new  item  (item  18)  was  added  to  enhance  content  validity.  A  number  of 
preceptors  suggested  that  a  subcompetency  covering  respect  for  the 
dignity  of  the  patient  had  been  overlooked.  The  total  number  of  items 
in  the  revised  instrument  was  40. 
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Validation  of  the  Revised  Instrument 

Fourth  Year  Physical  Therapy  Students 

Scores  on  the  revised  edition  of  the  Evaluation  of  Clinical 
Competence  for  each  group  of  students  studied  -  experienced  fourth 
year  PT  students  and  novice  third  year  PT  students  -  were  analyzed  to 
assist  in  determining  the  reliability  and  validity  of  the  instrument. 

To  determine  the  inter-rater  reliability  of  the  revised  ECC  with 
fourth  year  PT  students,  the  percentage  agreement  among  preceptors 
(when  the  not  observed  rating  was  removed  from  the  data)  was  tabulated 
for  each  of  the  40  subcompetencies  or  items  and  for  the  entire  instru¬ 
ment.  The  pattern  of  agreement  among  the  preceptors  on  the  40  items 
is  demonstrated  by  their  ranking  according  to  the  percentage  agreement 
results.  Table  5  indicates  for  each  item  the  number  of  times  the, 
"Not  Observed",  rating  was  used,  the  percentage  agreement,  and 
discrepant  pairs  of  ratings  contributing  most  to  the  percentage 
disagreement  among  the  preceptors. 

The  percentage  agreement  for  all  subcompetencies  ranged  from  80% 
to  31.8%.  The  mean  percentage  agreement  for  the  total  instrument  was 
55.6%,  ana  the  median  value  was  57.1%.  The  not  observed  rating  was 
used  more  than  half  of  the  time  for  two  items,  19  and  28,  which  were 
also  in  this  same  troublesome  category  in  the  first  edition  of  the  ECC. 

Percentage  agreement  of  items  on  the  first  edition  and  the 
revised  edition  was  compared.  Items  which  persisted  to  have  a  low 
percentage  of  agreement  pertain  to  displaying  interest  in  the 
specialty,  identifying  secondary  problems,  recording  the  initial 
evaluation  and  adhering  to  safety  precuations,  all  of  which  are 
extremely  important  to  the  practice  of  physical  therapy.  Items 
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Item 

16 

28 

14 

30 

13 

19 

27 

12 

31 

38 

3 

20 

18 

35 

7 

5 

11 

22 

37 

25 

36 

10 

1 

21 

26 

4 

6 

24 

29 

32 
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Table  5 


Subcompetencies  Ranked  by  Percentage  Agreement  Among 
Preceptors  on  the  Revised  Evaluation  of  Clinical 
Competence  with  Fourth  Year  PT  Students 


Frequency  of 

% 

%  Disagreement 

Description 

Not  Observed 

Agreement 

and  Cell 

tidies  treatment  area 
assists  others  implement  aspects 
of  patient  treatment 
observes  skillfully 
records  progress  notes 
treats  skillfully 
prepares  patient  for  discharge 
communicates  orally 
justifies  treatment  plan 
maintains  confidentiality 
identifies  own  strengths  and 
weaknesses 

conducts  patient  interview 
educates  patient/family 
ensures  dignity  of  patient 
assumes  appropriate  PT  roles 
uses  knowledge  for  planning 
evaluates  skillfully 
selects  treatment  procedures 
motivates  patient/family 
modifies  beavhior  to  meet 
suggestions 

schedules  patient  load 
asks  questions 
determines  treatment  goals 
uses  knowledge  for  evaluating 
instructs  patient/family 
develops  rapport  with  others 
selects  evaluation  procedures 
eases  patient's  apprehension 
assists  patient  implement  own 
treatment 

records  initial  evaluation 
is  reliable 


5 

80.0 

F 

15.0 

20 

80.0 

F 

20.0 

1 

79.2 

F 

16.7 

5 

70.0 

D/E/F 

10.0 

0 

68.0 

F 

20.0 

22 

66.7 

C 

33.3 

7 

66.7 

F 

22.2 

2 

65.2 

B 

13.0 

5 

65.0 

F 

30.0 

5 

65.0 

F 

25.0 

8 

64.7 

D/F 

17.6 

8 

64.7 

D 

17.6 

7 

61.1 

F 

38.9 

2 

60.9 

F 

26.1 

1 

58.3 

D 

25.0 

1 

58.3 

E 

20.8 

1 

58.3 

F 

16.7 

1 

58.3 

F 

33.3 

6 

57.9 

F 

26.3 

11 

57.1 

F 

35.7 

4 

57.1 

F 

28.6 

2 

56.5 

F 

17.4 

5- 

55.0 

F 

20.0 

1 

54.2 

F 

33.3 

6 

52.6 

F 

42.1 

1 

50.0 

D 

29.2 

3 

50.0 

F 

40.9 

11 

50.0 

F 

28.6 

0 

48.0 

F 

20.0 

2 

47.8 

F 

47.8 

. 
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Table  5  (continued) 


Item 

Frequency  of 
Description  Not  Observed 

% 

Agreement 

%  Disagreement 
and  Cell 

17 

ensures  safety 

0 

44.0 

F 

40.0 

23 

develops  rapport  with  patient 

2 

43.5 

F 

39.1 

33 

presents  professional  appearance 

2 

43.5 

F 

43.5 

39 

uses  self-assessment  to  improve 

8 

41.2 

F 

35.3 

15 

modifies/progresses  treatment 

3 

40.9 

F 

36.4 

9 

identifies  secondary  problem 

4 

38.1 

D 

23.8 

2 

gathers  data  from  patient  chart 

9 

37.5 

F 

37.5 

8 

identifies  primary  problems 

1 

37.5 

F 

37.5 

40 

seeks  resources  to  increase 
knowledge 

9 

37.5 

F 

31.3 

34 

displays  interest 

3 

31.8 

F 

40.9 

Note:  n=25 
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continuing  to  show  a  high  percentage  agreement  were  related  to  pre¬ 
paring  the  patient  for  discharge,  educating  the  patient/family, 
maintaining  confidentiality  and  tidying  the  treatment  area. 

Table  6  also  presents  the  Kappa  coefficient  for  each  item.  The 
median  Kappa  coefficient  was  .32.  Even  though  Kappa  coefficients 
demonstrate  only  fair  agreement  between  preceptors,  27  items  showed 
significance  at  or  below  the  .05  probability  level. 

Cells  in  the  crosstabulation  tables  were  surveyed  to  determine 
which  discrepant  rating  pairs  contributed  to  the  percentage  disagree¬ 
ment  among  the  preceptors.  In  line  with  expectations,  results 
verified  that  the  percentage  disagreement  was  indeed  low  in  cells  A 
(1-2  rating  pair),  B  (1-3  rating  pair),  C  (1-4  rating  pair),  D  (2-3 
rating  pair)  and  E  (2-4  rating  pair),  where  rating  pairs  cross 
critical  ratings.  Furthermore,  percentage  disagreement  was  high  in 
cell  F  (3-4  rating  pair),  which  was  expected  particularly  in  this 
final  clinical  placement  prior  to  graduation  when  the  vast  majority  of 
students  should  achieve  a  competent  standard  of  performance.  Table  5 
does,  in  fact,  illustrate  that  cell  F  contributed  highly  to  the 
percentage  disagreement. 

Disagreement  attributable  to  cell  F  was  highest  among  the  items 
under  the  major  competency  areas  of  professional  behavior  and  communi¬ 
cation  with  patient/family.  The  mean  number  of  F  cell  rating  pairs 
for  these  items  is  6.9  and  6.2  respectively.  By  contrast,  the  mean 
number  of  D  cell  rating  pairs  was  highest  for  items  under  program 
planning  at  3.7  and  documentation  at  3.0.  When  the  discrepant  pairs 
in  the  remaining  cells  (A,  B,  C  and  E)  were  considered  together  and 
averaged,  the  items  under  program  planning  and  documentation  were 
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Table  6 


Kappa  Coefficients  Among  Preceptors  for  each  Subcompetency  in 
the  Revised  Evaluation  of  Clinical  Competence  with 
Fourth  and  Third  Year  PT  Students 


4th  Year  Kappa  3rd  Year  Kappa 
Item  Coefficient  Coefficient 

4th  Year  Kappa  3rd  Year  Kappa 
Item  Coefficient  Coefficient 

1 

.353** 

.068 

21 

.198 

.270** 

2 

.226** 

.501** 

22 

.324* 

.319** 

3 

.448** 

.812** 

23 

.010 

.319* 

4 

.188 

.170 

24 

.342** 

.528** 

5 

.286* 

.283 

25 

.478** 

.588** 

6 

.234* 

.285 

26 

.269* 

.537** 

7 

.356** 

.319* 

27 

.420** 

.241** 

8 

.023 

.083 

28 

.782** 

.495** 

9 

.183* 

.095 

29 

.207 

.287** 

10 

.359** 

.306* 

30 

.570** 

.398** 

11 

.274 

.259 

31 

.347* 

.361** 

12 

.314 

.376* 

32 

.111 

0. 

13 

.506** 

.443** 

33 

-.003 

.33 

14 

.616** 

.186 

34 

-.038 

.299* 

15 

.105 

.467** 

35 

.299* 

.163 

16 

.549** 

.55** 

36 

.330** 

.194 

17 

.168 

-.079 

37 

.371** 

.047 

18 

.387** 

.394** 

38 

.457** 

.544** 

19 

.664** 

.658** 

39 

.130 

.466** 

20 

.462** 

.344** 

40 

.182* 

.251** 

★ 


£  <  .05 

£  £  .01 
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again  highest  at  3.3  and  3.0  respectively.  Where  discrepant  rating 
pairs  exist,  there  was  no  apparent  trend  whereby  the  first  preceptors 
consistently  chose  the  higher  or  lower  rating  of  the  pair. 

Since  disagreement  was  largely  attributed  to  3-4  rating  pairs 
(cell  F),  both  of  which  measure  competence  at  acceptable  levels,  these 
ratings  were  therefore  treated  as  one  competency  rating.  The 
percentage  agreement  was  retabulated  for  the  resultant  3-point  scale. 

Table  7  presents  the  percentage  agreement  of  the  3-point  scale 
for  the  40  items  by  the  major  competency  areas  and  their  ranking  from 
high  to  low  percentage  agreement.  The  mean  percentage  agreement  for 
the  total  instrument  using  a  3-point  scale  improved  considerably  to 
81.7%  with  the  percentage  agreement  of  items  ranging  from  100%  to 
52.4%. 

Items  which  showed  a  high  percentage  agreement  (greater  than  90% 
agreement),  listed  in  order  were  those  dealing  with  ensuring  dignity 
of  the  patient,  assisting  health  care  personnel  implement  aspects  of 
PT  treatment,  observing  skillfully,  being  reliable,  respecting 
confidentiality,  developing  rapport  with  team  members,  scheduling 
patient  load,  motivating  patients/family,  easing  patient's 
apprehension  and  identifying  own  strengths  and  weaknesses.  Items 
which  showed  a  low  percentage  agreement  (less  than  70%  agreement), 
listed  from  the  lowest  to  highest  values,  were  those  related  to 
identifying  secondary  problems,  selecting  evaluation  procedures,  using 
knowledge  for  treatment  planning,  preparing  patient  for  discharge, 
recording  the  initial  evaluation  and  displaying  interest  in  the 
special ity. 

A  general  impression  of  agreement  is  obtained  by  looking  at  the 
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Table  7 

Percentage  Agreement  Among  Preceptors  on  a  3-point  Scale 
for  the  Revised  Evaluation  of  Clinical  Competence  with 
Fourth  Year  and  Third  Year  PT  Students 


Item 

Weight 

Fourth  Year 
%  Agreement 
(3-point  scale) 

Rank 

Third  Year 
%  Agreement 
(3-point  scale) 

Rank 

Patient 

Evaluation 

1 

3 

75.0 

23 

36.8 

31 

2 

2 

75.0 

23 

83.3 

10 

3 

3 

82.4 

18 

88.2 

5 

4 

3 

66.7 

29 

52.2 

28 

5 

3 

70.8 

26 

65.2 

22 

6 

2 

90.9 

8 

73.7 

j_7 

X  =  76.8 

5 

X  =  66.6 

4 

Program  Planning 


7 

3 

66.7 

29 

63.6 

24 

8 

3 

75.0 

23 

65.2 

22 

9 

2 

52.4 

30 

41.2 

30 

10 

2 

73.9 

24 

60.0 

26 

11 

3 

75.0 

23 

60.9 

25 

12 

2 

73.9 

24 

63.6 

24 

X  =  69.5 

7 

X  =  59.1 

6 

Implementation  of  Treatment 


13 

3 

88.0 

11 

76.2 

15 

14 

3 

95.8 

2 

76.2 

15 

15 

3 

77.3 

21 

68.4 

20 

16 

1 

95.0 

4 

75.0 

16 

17 

3 

84.0 

16 

71.4 

18 

18 

3 

100.0 

1 

85.0 

9 

19 

2 

66.7 

29 

100.0 

J_ 

X  =  86.7 

2 

X  =  78.9 

2 
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Table  7  (continued) 


Item 

Weight 

Fourth  Year 
%  Agreement 
(3-point  scale) 

Rank 

Third  Year 
%  Agreement 
(3-point  scale) 

Rank 

Communication 

with  Patient/Family 

20 

2 

76.5 

22 

60.0 

26 

21 

2 

87.5 

12 

60.9 

25 

22 

2 

91.7 

7 

68.2 

21 

23 

2 

82.6 

17 

81.0 

12 

24 

2 

78.6 

20 

60.0 

26 

X  =  83.4 

4 

X  =  66.0 

7 

Communication  with  Health  Care  Personnel 

25 

1 

92.9 

6 

75.0 

16 

26 

2 

94.7 

5 

85.7 

8 

27 

2 

88.9 

10 

58.8 

27 

28 

2 

100.00 

J_ 

50.0 

29 

X  =  94.1 

1 

X  =  67.4 

5 

Documentation 


29 

3 

68.0 

28 

65.0 

23 

30 

3 

80.0 

J_9 

78.9 

J3 

X  =  74.0 

6 

X  =  72.0 

3 

. 
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Table  7  (continued) 


Item 

Weight 

Fourth  Year 
%  Agreement 
(3-point  scale)  Rank 

Third  Year 
%  Agreement 
(3-point  scale) 

Rank 

Professional  Behavior 

31 

3 

95.0 

4 

94.1 

2 

32 

2 

95.7 

3 

90.9 

4 

33 

2 

87.0 

13 

91.3 

3 

34 

3 

72.7 

25 

86.4 

7 

35 

2 

87.0 

13 

81.0 

12 

36 

3 

85.7 

14 

81.0 

12 

37 

2 

84.2 

15 

87.0 

6 

38 

2 

90.6 

9 

81.8 

11 

39 

2 

76.5 

22 

77.8 

14 

40 

2 

68.8 

27 

69.2 

T9 

X  =  84.3 

3 

X  =  84.1 

1 
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mean  percentage  agreement  within  the  major  competency  areas  and  their 
rankings  in  Table  7.  Items  dealing  with  communication  with  health 
care  personnel  and  implementation  of  treatment,  tended  to  show  the 
highest  percentage  agreement  among  the  preceptors  and  were  followed 
closely  by  items  related  to  professional  behavior  and  communication 
with  patient/family.  Whereas,  items  pertaining  to  program  planning, 
documentation  and  patient  evaluation  tended  to  show  the  lowest  agree¬ 
ment  among  the  preceptors. 

The  means  of  students'  raw  scores  for  each  of  the  40  items 

ranged  from  3.80  to  .52,  out  of  a  possible  of  4.00.  Although  the 
first  preceptors  tended  to  have  higher  means  and  less  variance  than 
the  second  preceptors,  there  was  a  significant  difference  between  the 
means  for  only  five  items:  19,  28,  31,  36  and  38. 

The  students'  raw  scores  were  transformed  using  weights  (Table 
7)  to  produce  for  each  student  a  total  scale  score  and  seven  subscale 
scores  for  each  of  the  major  competency  areas.  Table  8  presents  the 
intraclass  coefficients  for  the  total  scale  score  and  the  seven 
subscale  scores  along  with  their  F  values  and  probability  levels.  The 
reliability  of  the  ECC  was  .591  for  fourth  year  PT  students. 

The  total  scale  scores  ranged  from  4.00  to  2.23  with  a  mean 

scale  score  of  3.22  by  the  first  preceptors  and  with  a  mean  scale 
score  of  3.19  by  the  second  preceptor.  Although  the  second  preceptor 
tended  to  give  a  greater  variation  of  scores,  there  was  not  a  signifi¬ 
cant  difference  between  the  mean  scale  scores  given  by  the  two 

preceptors  (p  =  .74) . 

Results  indicate  that  concurrent  validity  of  the  ECC  was 

substantial.  Table  9  shows  correlations  between  the  ECC  total  scores 
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Table  8 


Intraclass  Coefficients  and  Results  of  One-Way  Analysis  of  Variance 
with  Repeated  Measures  for  the  Total  Score  and  Subscale  Scores  on 
the  Revised  Evaluation  of  Clinical  Competence  with 
Fourth  and  Third  Year  PT  Students 


Fourth 

Year3 

Third  Yearb 

Intraclass0 

Coefficients 

F 

0,24) 

P 

Intraclass0 

Coefficients 

F 

0,22) 

P 

patient  evaluation 

— 

.0 

1.00 

.566 

.03 

.86 

program  pi anning 

.543 

.02 

.89 

.451 

.02 

.88 

implementation  of 
treatment 

.607 

.04 

.85 

.403 

.03 

.86 

communication 
patient/fami  ly 

.286 

.16 

.70 

.317 

.68 

.42 

communication  health 
care  personnel 

.336 

.81 

.38 

.521 

.20 

.  66 

documentation 

.203 

1.81 

.29 

.308 

.13 

.72 

professional  behavior 

.092 

2.71 

.11 

.257 

.66 

.42 

total  scale  score 

.591 

.11 

.74 

.624 

.70 

.41 

a  n=25 
b  n=23 

c  adjusted  reliability  values  used 
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Table  9 


Correlation^  of  Total  Scale  Scores  on  the  Revised  Evaluation 
of  Clinical  Competence  with  Hiring  Ratings  and 
UA  Performance  Scores  for  the  Fourth  Year 
PT  Students^ 


Hiring  Ratings 
Preceptor  1  Preceptor  2 

UA  Scores 
Preceptor  1  only 

Total  ECC  Scores 

Preceptor  1 

.597 

.493 

Preceptor  2 

.655 

— 

Pearson  product-moment  correlations 


b 


n=25 
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and  (1)  preceptor  hiring  ratings,  and  (2)  UA  performance  scores.  The 
hiring  rating  form  demonstrated  good  reliability  with  fourth  year  PT 
students  over  the  two  preceptors  at  .842. 

Validation  of  the  Revised  Instrument 

Third  Year  Physical  Therapy  Students 

To  determine  the  inter-rater  reliability  of  the  revised  ECC  with 
third  year  PT  students,  the  percentage  agreement  among  the  preceptors 
(when  the  not  observed  rating  was  removed  from  the  data)  was  tabulated 
for  each  of  the  40  subcompetencies  and  for  the  entire  instrument.  The 
pattern  of  agreement  among  the  preceptors  on  the  40  item  is  demon¬ 
strated  by  their  ranking  according  to  the  percentage  agreement 
results.  Table  10  indicates  for  each  subcompetency  the  number  of 
times  the  not  observed  rating  was  used,  the  percentages  of  agreement 
and  discrepant  pairs  of  rating  contributing  most  to  the  percentage 
disagreement  among  the  preceptors. 

The  percentage  agreement  for  all  items  ranged  from  88.2%  to 
33.3%.  The  mean  percentage  agreement  for  the  entire  instrument  was 
54.8%  and  the  median  was  56.4%.  The  not  observed  rating  was  used  more 
than  half  of  the  time  for  seven  items:  19,  24,  25,  26,  28,  38  and 
39.  Only  items  19  and  28  were  also  in  this  same  category  for  fourth 
year  students. 

Percentage  agreement  of  fourth  year  students  and  third  year 
students  was  compared.  Items  with  high  percentages  of  agreement 
common  to  both  groups  of  students  were  items  18,  3,  38,  12,  19,  13, 
16.  Items  with  low  percentages  of  agreement  common  to  both  groups  of 
students  were  items  17,  9,  8,  40. 
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Table  10 


Subcompetencies  Ranked  by  Percentage  Agreement  Among 
Preceptors  on  the  Revised  Evaluation  of  Clinical 
Competence  with  Third  Year  PT  Students 


Item 

Frequency  of 
Description  Not  Observed 

% 

Agreement 

%  Disagreement 
and  Cell 

3 

conducts  patient  interview 

6 

88.2 

D 

11.8 

33 

presents  professional 
appearance 

0 

73.9 

F 

17.4 

13 

treats  ski  1 lfully 

2 

66.7 

D 

23.8 

16 

tidies  treatment  area 

11 

66.7 

E 

16.7 

19 

prepares  patient  for  discharge 

20 

66.7 

F 

33.3 

38 

identifies  own  strengths  and 
weaknesses 

12 

63.6 

F 

18.2 

12 

justifies  treatment  plan 

1 

63.6 

D 

27.3 

15 

modifies/progresses  treatment 

4 

63.2 

D 

15.8 

25 

schedules  patient  load 

15 

62.5 

B/E/F 

12.5 

5 

evaluates  skillfully 

0 

60.9 

D 

30.4 

11 

selects  treatment  procedures 

0 

60.9 

D 

30.4 

24 

assists  patient  implement 
own  treatment 

13 

60.0 

D 

30.0 

18 

ensures  patient  dignity 

3 

60.0 

F 

25.0 

7 

uses  knowledge  for  planning 

1 

59.1 

D 

27.3 

31 

maintains  confidentiality 

6 

58.8 

F 

35.3 

2 

gathers  data  from  patient 
chart 

11 

58.0 

F 

25.0 

30 

records  progress  notes 

4 

57.9 

F 

21.1 

26 

develops  rapport  with  others 

16 

57.1 

F 

28.6 

14 

observes  skillfully 

2 

57.1 

D/F 

19.0 

23 

develops  rapport  with 
patients 

2 

57.1 

F 

23.8 

39 

uses  self-evaluation  to 
improve 

14 

55.6 

F 

22.2 

10 

determines  treat  goals 

3 

55.0 

D 

30.0 

34 

displays  interest 

1 

54.5 

F 

31.8 

6 

eases  patient's  apprehension 

4 

52.6 

D 

26.3 

4 

selects  evaluation  procedures 

0 

52.2 

D 

30.4 

28 

assists  others  implment  aspect 
of  patient  treatment 

21 

50.0 

D 

4.3 

29 

records  initial  evaluation 

3 

50.0 

D 

20.0 

21 

instructs  patient/family 

0 

47.8 

D/F 

13.0 

. 
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Table  10  (continued) 


Item 

Frequency  of 
Description  Not  Observed 

% 

Agreement 

%  Disagreement 
and  Cell 

35 

assumes  appropriate  PT  roles 

2 

47.6 

F 

33.3 

36 

asks  questions 

2 

47.6 

F 

33.3 

20 

educates  patient/family 

8 

46.7 

D 

6.7 

22 

motivates  patient/family 

1 

45.5 

F 

22.7 

32 

is  reliable 

1 

45.5 

F 

45.5 

37 

modifies  behavior  to  meet 
suggestions 

0 

43.5 

F 

43.5 

8 

identifies  primary  problems 

0 

43.4 

D 

30.4 

9 

identifies  secondary  problems 

6 

41.2 

D 

41.2 

27 

communicates  orally 

6 

41.2 

E 

23.5 

40 

seeks  resources  to  increase 
knowledge 

10 

38.5 

F 

30.8 

1 

uses  knowledge  of  evaluating 

4 

36.8 

D 

47.4 

17 

ensures  safety 

2 

33.3 

D 

28.6 

Note:  n=23 
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Table  6  presents  the  Kappa  coefficients  for  each  item.  The 
median  Kappa  coefficient  was  .32.  Although  the  Kappa  coefficients  was 
low,  26  items  showed  significance  at  or  below  a  probability  level  of 

.05. 

Cells  in  the  crosstabulation  tables  were  scrutinized  to  deter¬ 
mine  which  discrepant  rating  pairs  contribute  to  the  percentage 
disagreement  among  the  preceptors.  Table  10  indicates  that  both  cells 
D  (2-3  rating  pairs)  and  F  (3-4  rating  pairs)  were  responsible  for  the 
large  percentage  of  the  disagreement.  Disagreement  attributable  to 
cell  F  was  highest  among  the  items  under  the  major  competency  areas  of 
professional  behavior  and  documentation  with  the  mean  number  of  F  cell 
ratings  pairs  at  5.9  and  3.5  respecti vely.  By  contrast,  the  mean 
number  of  D  cell  rating  pairs  was  highest  for  items  under  program 
planning  at  6.5  and  patient  evaluation  at  5.33.  When  the  discrepant 
pairs  in  the  remaining  cells  were  considered  together  and  averaged, 
the  items  under  documentation  and  communication  with  patient/family 
were  highest  at  3.0  and  2.8  respecti vely. 

Once  again,  the  3  and  4  ratings  were  treated  as  one  competency 
rating  and  the  percentage  agreement  was  retabulated  for  the  resultant 
3-point  scale.  Table  7  presents  the  percentage  agreement  of  the 
3-point  scale  for  the  40  items  by  the  major  competency  area  and  their 
ranking  from  high  to  low  percentage  agreement.  The  mean  percentage 
agreement  for  the  instrument  using  the  3-point  scale  improved  to  73.2% 
with  percentage  agreements  of  items  ranging  from  100%  to  36.8%. 

Items  which  showed  a  high  percentage  agreement  (greater  than  90% 
agreement),  listed  in  order,  are  those  pertaining  to  respecting 
confidentiality,  presenting  a  professional  appearance,  and  being 


' 


%  - 


79 


reliable.  Items  which  showed  a  low  percentage  agreement  (less  than 
60%  agreement),  listed  from  lowest  to  highest  values,  were  those 
related  to  communicating  orally,  selecting  evaluation  procedures, 
identifying  secondary  problems  and  using  knowledge  for  evaluating. 
Items  19  and  28  were  troublesome  because  the  not  observed  rating  was 
frequently  used. 

A  general  impression  of  agreement  is  obtained  by  looking  at  the 
rankings  of  the  mean  percentage  agreement  of  the  items  within  the 
major  competency  areas  in  Table  7.  Items,  dealing  with  professional 
behavior  and  implementation  of  treatment,  tended  to  have  the  highest 
percentage  agreement  among  the  preceptors.  Whereas,  items  pertaining 
to  communication  with  family/family  and  program  planning  tended  to 
have  the  lowest  agreement  among  the  preceptors. 

The  means  of  the  students'  raw  scores  for  each  item  ranged  from 
3.90  to  .391.  Even  though  the  first  preceptors  tended  to  have  higher 
means  and  less  variance  than  the  second  preceptors,  there  was  a 
significant  difference  between  the  means  for  only  three  items:  26,  38 
and  39.  These  items  were  different  from  those  identified  with  the 
fourth  year  students. 

The  students'  raw  scores  were  transformed  using  the  weights 
(Table  7)  to  produce  for  each  student  a  total  scale  score  and  seven 
subscale  scores.  Table  8  presents  the  intraclass  coefficients  for  the 
total  scale  score  and  the  seven  scale  scores  along  with  their  F  value 
and  probability  levels.  The  reliability  of  the  ECC  total  scale  scores 
was  .620. 

The  total  scale  scores  for  the  third  year  students  ranged  from 
3.52  to  1.80  with  a  mean  scale  score  of  2.72  from  the  first  preceptors 
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and  with  a  mean  scale  score  of  2.79  from  the  second  preceptors. 
Overall,  the  second  preceptors  tended  to  give  slightly  higher  total 
scores  with  a  wider  variance  of  scores,  but  this  difference  between 
the  preceptors  was  not  significant  (p  =  .41). 

Third  year  results  also  indicated  substantial  concurrent 
validity  of  the  ECC.  Table  11  shows  the  correlations  between  ECC 
total  scale, scores  and  (1)  the  preceptor's  hiring  ratings,  and  (2)  the 
UA  performance  scores.  The  reliability  coefficient  of  the  hiring 
rating  with  third  year  PT  students  was  .512  over  the  two  preceptors. 

Results  in  Table  12  support  construct  validity  of  the  ECC.  The 
prediction  that  experienced  students  would  perform  better  than  novice 
students  was  verified  by  the  significant  difference  found  between  mean 
ECC  total  scale  scores  awarded  to  the  two  groups  of  students  by  the 
two  preceptors  [F  (1,97)  =  26.06,  p  =  .000]. 

Reactions  of  the  Preceptors  and  Students  to  the  Instrument 

The  total  number  of  replies  to  the  questionnaire  regarding 
preceptors'  reactions  to  the  ECC  was  34  or  81%. 

The  reactions  of  preceptors  to  the  content,  administration, 
format  and  effects  of  the  ECC  tended  to  be  favorable,  despite  the 
extra  burden  placed  on  them  to  observe  daily  and  to  complete  a  longer 
and  more  detailed  evaluation  than  ever  before.  In  fact,  63%  of 
preceptors  viewed  the  administration  procedures  and  instructions  for 
the  study  positively. 

The  majority  of  preceptors  felt  that  the  number  of  subcompeten¬ 
cies  included  in  the  ECC  was  about  right  (82%  of  preceptors)  and  that 
content  was  relevant  (85%).  The  four  standards  of  performance  for 
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Table  11 


Correlation3  of  Total  Scale  Scores  on  the  Revised  Evaluation 
of  Clinical  Competence  with  Hiring  Ratings  and 
UA  Performance  Scores  for  the  Third  Year 
PT  Students^ 


Hiring  Ratings 
Preceptor  1  Preceptor  2 

UA  Scores 
Preceptor  1  only 

Total  ECC  Scores 

Preceptor  1 

.575 

.  666 

Preceptor  2 

.487 

— 

Pearson  product-moment  correlations 


b 


n=23 
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Table  12 


Results  of  One-Way  Analysis  of  Variance  for  the  Total  Scores 
Awarded  Fourth  Year  and  Third  Year  PT  Students  on  the 
Evaluation  of  Clinical  Competence 


F 

x  s2  (1,91)  p 


Year 

Fourth 


3.215  .189  26.06  .000 


Thi  rd 


2.758 


.208 
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each  subcompetency  were  considered  about  right  (91%)  and  descriptive 
enough  to  judge  a  student's  performance  (71%).  The  qualitative 
differences  between  each  of  the  four  standards  were  perceived  to  be 
about  right  (68%),  although  16%  felt  that  the  differences  were  too 
large  and  another  16%  felt  that  the  differences  were  too  small.  The 
ability  of  the  ECC  to  evaluate  a  student  objectively  was  viewed  as 
good  or  adequate  by  91%  of  the  preceptors.  The  format  of  the  form  and 
its  ease  of  completion  were  liked  by  74%  of  the  preceptors;  however, 
55%  of  the  preceptors  would  have  liked  space  provided  for  subjective 
interpretation  of  the  student's  performance. 

Daily  observations  of  students  were  viewed  negatively  by  only 
21%  of  the  preceptors.  All  subcompetencies  were  usually  easy  to 
observe  (76%  to  94%  of  preceptors),  except  those  related  to  communi¬ 
cation  with  other  health  care  personnel  which  were  seldom  easy  to 
observe  (47%).  Although  82%  of  preceptors  felt  that  the  final  rating 
should  be  determined  by  considering  the  daily  ratings  given  throughout 
the  placement,  39%  of  preceptors  were  negative  toward  rating  perfor¬ 
mance  on  the  form  on  a  daily  basis. 

The  ECC  was  perceived  to  have  a  positive  impact  on  the 
preceptor's  skill  in  evaluating  students  (88%)  and  on  the  level  of 
preceptor-student  interaction  (82%).  A  negative  effect  was  noted  by 
preceptors  regarding  the  students'  high  levels  of  anxiety  elicited  by 
being  observed  by  two  preceptors  (55%). 

The  total  number  of  replies  to  the  questionnaire  regarding  the 
students'  reactions  to  the  ECC  was  22  or  40%. 

The  reactions  of  the  students  to  the  content,  administration, 
format  and  effects  of  the  ECC  tended  to  be  favorable,  despite  the 
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anxiety  provoked  by  being  observed  daily  by  two  preceptors.  Daily 
observations  were  viewed  negatively  by  only  18%  of  the  students. 
Furthermore,  only  10%  of  the  students  viewed  the  administration 
procedures  for  the  study  negatively.  The  anxiety  provoked  was 
perceived  to  influence  their  clinical  performance  in  different  ways. 
Nineteen  percent  of  students  felt  that  the  anxiety  had  a  positive 
influence,  38%  felt  that  it  had  a  negative  influence,  and  43%  were 
unsure. 

The  ability  of  the  ECC  to  evaluate  a  student  objectively  was 
considered  good  by  81%  of  the  students.  Seventy-one  percent  and  64% 
of  the  students  felt  that  the  ECC  had  a  positive  effect  on  the 
preceptors'  skills  in  evaluating  students  on  preceptor-student  inter¬ 
actions,  respectively.  Eighty-two  percent  of  students  felt  that  the 
final  rating  should  be  determined  by  considering  the  daily  ratings 
given  throughout  the  placement.  Only  36%  of  the  students  felt  that 
space  should  be  provided  for  comments. 


Chapter  V 


DISCUSSION 

Instrument  Development  Phase 


One  major  contribution  of  the  present  study  is  the  description 
of  the  process  involved  in  developing  a  clinical  evaluation  instru¬ 
ment.  Content  validity  is  the  first  concern  in  the  development  of  a 
clinical  evaluation  because  of  its  intended  use.  An  instrument,  which 
is  intended  to  determine  if  a  PT  student  can  perform  entry-level 
clinical  competencies  and,  subsequently,  to  make  decisions  regarding 
student  promotion  and  graduation,  must  have  an  obvious  link  to  the  job 
to  which  the  students  will  graduate. 

The  determination  of  clinically  significant  competencies  needed 
for  development  of  such  an  instrument  can  be  based,  in  part,  on 
response  ratings  of  the  importance  of  competencies  outlined  in  the 
questionnaire.  The  verification  of  competencies  and  their  relative 
importance  obtained  in  this  way  provided  a  sound  basis  for  sampling 
competencies  for  evaluation.  Logically,  all  important  competencies 
constituting  the  domain  of  clinical  competence  must  be  included  in  the 
instrument,  thereby  ensuring  content  validity. 

The  study  has  also  shed  some  light  on  information  that  is  most 
needed  for  defining  the  construct  of  clinical  competence.  Identifica¬ 
tion  of  clinical  competence  is  most  important  for  structuring  the 
operations  of  clinical  supervision  and  evaluation.  Preceptors  are 
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informed  of  what  to  expect  from  students  and  by  what  standards  to 
judge  the  students'  performances.  Students  are  informed  as  to  what 
competencies  that  they  are  expected  to  perform  and  to  what  standards. 

In  the  questionnaire  the  respondents  rated  the  importance  of 
competencies  according  to  their  interpretation  of  its  importance  to  PT 
practice.  In  these  ratings,  three  major  competency  areas  -  patient 
evaluation,  program  planning  and  implementation  of  treatment  -  ranked 
highest,  which  may  reflect  an  attitude  on  the  part  of  practitioners 
that  technical  knowledge  and  skills  have  a  more  direct  impact  on 
treatment  of  patients  than  does  the  affective  component  of  competence. 

In  contrast  to  highly  ranked  competencies,  five  major  competency 
areas  -  documentation,  communication  with  health  care  personnel, 
professional  behavior  and  professional  growth  and  management  skills  - 
were  ranked  comparati vely  lower.  The  inference  may  be  that  those 
areas  not  having  a  direct  implication  for  treatment  of  the  patient  are 
not  considered  as  important.  Furthermore,  subcompetencies  descriptive 
of  menial  or  bothersome  tasks,  e.g.  scheduling,  tidiness  of  treatment 
area,  supervision  of  support  personnel,  and  adherence  to  operational 
procedures,  were  also  not  viewed  important. 

In  a  client-centered  profession  such  as  PT  the  lower  ratings  of 
importance  given  to  subcompetencies  related  to  communication  with  the 
patient  and  family  about  their  evaluation  findings,  treatment  plan  and 
physical  status  and  treatment  seems  inappropriate.  This  finding  is 
particularly  odd  since  the  area  of  communication  with  patient  and 
family  was  the  fourth  top  ranking  major  competency  area.  Competence 
in  this  area  is  perceived  as  ideal  but  is  not  necessarily  being 
pursued  in. practice. 
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The  lower  importance  value  placed  on  these  affective  subcom¬ 
petencies  may  reflect  the  curriculum  that  produced  the  practitioner 
rather  than  the  job  requirements.  The  professional  training  programs 
are  lengthy  and  demanding  with  emphasis  placed  on  developing  technical 
knowledge  and  skills.  Interpersonal  and  communication  skills  are  not 
explicitly  included  in  the  curricula  of  many  programs.  This  result 
may  suggest  further  that  the  importance  of  affective  behaviors  are 
also  not  being  reinforced  with  students  in  the  clinical  setting. 

Factor  analysis  of  medical  evaluations  has  revealed  two  factors 
of  the  construct  of  clinical  competence  (DiMatteo  and  DiNicola,  1981; 
Davidge,  Davis  and  Hull,  1980;  Oratio,  1976),  which  for  the  most  part 
are  termed,  technical  skills  and  interpersonal  skills.  The  emergence 
of  the  i nterpersonal  factor  suggests  that  affective  behaviors  are 
critical  to  therapeutic  effectiveness.  Again,  the  question  arises 
about  the  kinds  of  professional  training  required  for  competent  thera¬ 
peutic  interactions.  Courses  in  group  dynamics  and  communication 
skills  may  enhance  self-awareness  and  interpersonal  relationships 
during  therapy. 

Another  explanation  for  lower  importance  ratings  of  subcom¬ 
petencies  related  to  interpersonal  skills  may  be  because  practitioners 
are  less  sure  such  subcompetencies  should  be  used  as  criteria  for 
student  evaluation.  In  a  study  on  affective  education,  Lacefield 
(1981)  found  that  allied  health  professionals  perceived  affective 
skills  to  be  a  required  area  of  competence  for  effective  practice; 
paradoxically,  however,  they  did  not  perceive  them  to  be  suitable  for 
inclusion  in  formal  education  and  evaluation  of  students. 

Caution  must  be  exercised  when  compiling  a  list  of  competencies 
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for  evaluation  purposes  from  a  list  of  competencies  rated  by 
practitioners  according  to  their  importance  in  relation  to  their  own 
professional  practice.  A  subcompetency  with  minor  importance  must  be 
considered  in  relation  to  its  value  to  all  aspects  of  professional 
practice  and  growth.  Likewise,  competencies  appropriate  for  entry- 
level  practice  may  be  different  from  those  appropriate  for  experienced 
practitioners  working  in  diverse  roles,  such  as  in  education,  adminis¬ 
tration,  consultation  and  community  services.  The  wide  variation  in 
ratings  of  importance  may  give  some  indication  of  the  diverse  roles 
and  functions  assumed  by  physical  therapists. 

The  subcompetencies  related  to  education  of  the  patient  and 
family  regarding  his/her  physical  status  and  treatment  was  retained  in 
the  list  of  competencies  to  be  evaluated  by  the  instrument,  despite 
its  low  importance  rating.  It  is  perceived  to  be,  at  the  least,  a 
behavior  indicative  of  interpersonal  communications  conducive  to 
meeting  the  psychological  needs  of  the  patient  and  family  and  thereby 
conducive  to  a  good  physical  outcome  in  therapy. 

In  conclusion,  the  current  findings,  combined  with  those  of  past 
research,  may  be  instrumental  in  studying  the  subcompetencies  compris¬ 
ing  the  construct  of  clinical  competence  and  their  evaluation. 

Instrument  Validation  Phase 


Evaluation  of  the  Instrument 

Testing  the  ECC  on  a  sample  of  students  representative  of  the 
population,  with  which  the  final  edition  of  the  ECC  will  be  used,  was 
essential  to  making  modifications  prior  to  its  implementation.  The 
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sample,  by  necessity,  was  small  (ji=29),  due  to  small  class  sizes 
(n=35)  in  the  PT  program  at  the  UA.  Hence,  sample  size  limited  the 
interpretation  of  the  results. 

Subcompetencies  considered  important  to  the  evaluation  of 

students'  readiness  to  practice  PT  were  refined  and  used  to  construct 
the  revised  ECC.  Nine  subcompetencies  were  deleted  in  the  revised 
ECC,  because  they  contributed  minimally  to  the  evaluation  of 

students.  Specifically,  four  subcompetencies  were  deleted  because 
they  were  not  regularly  observed  by  preceptors  in  the  clinical  setting 
hence  could  not  be  evaluated.  Three  subcompetencies  of  low  importance 
were  dropped  to  shorten  the  ECC  to  a  more  practical  length  without 
jeopardizing  the  content  validity.  Another  two  subcompetencies  were 
omitted  because  the  content  was  dealt  with  more  specifically  under 
other  subcompetencies. 

The  greatest  disparity  in  the  ratings  between  the  preceptor 

pairs  was  from  items  considered  highly  important  to  PT  practice.  For 

example,  items  related  to  safety,  selecting  treatment  procedures  and 
recording  initial  evaluations  showed  poor  agreement.  This  result  may 
be  due  to  the  fact  that  important  subcompetencies  can  be  attained  by 
alternative  judgements  and  approaches  with  comparable  outcomes.  The 
important  items  susceptible  to  varied  clinical  judgements  were  worded 
more  specifically  to  narrow  the  interpretation  by  preceptors  to  the 
observable  behaviors  most  frequently  displayed  by  students. 

The  best  agreement  between  ratings  of  the  preceptor  pairs  was 
for  items  under  the  major  competency  areas  of  patient  evaluation  and 
implementation  of  treatment.  Examples  of  items  with  good  agreement 
were  those  pertaining  to  tidying  the  treatment  area,  selecting  evalua- 
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tion  procedures,  and  preparing  the  patient  for  discharge.  These  items 
likely  involve  more  precise  judgements  possible  in  the  technical  skill 
area  of  competence. 

Although  the  percentage  of  agreement  among  preceptors  for  each 
individual  item  in  the  ECC,  was  relatively  low  when  using  a  4-point 
scale,  agreement  among  preceptors,  expressed  as  intraclass  reliability 
coefficients,  was  good  when  items  were  pooled  to  obtain  a  total  scale 
score.  The  reliability  of  the  entire  instrument  was  .704. 

Validation  of  the  Revised  Instrument 

Reliability  of  the  Individual  Items 

The  percentage  agreement  among  the  preceptors  for  the  forty 
items  in  the  revised  ECC  was  relatively  low  using  a  4-point  scale. 
Also,  the  Kappa  coefficients  of  agreement  for  each  individual  item 
rated  on  a  4-point  scale  were  mainly  in  the  .21  to  .40  range,  which  is 
a  fair  level  of  agreement  according  to  the  classification  of  observer 
agreement  for  nominal  data  suggested  by  Landis  and  Koch  (1977).  Even 
though  the  Kappa  coefficients  of  the  items  showed  only  fair  agreement, 
two  thirds  of  the  items  had  significant  reliability  coefficients  at  or 
below  the  .05  probability  level.  A  significant  reliability  value 
indicates  that  the  amount  of  agreement  between  the  two  preceptors 
would  occur  by  chance  at  or  less  than  five  percent  of  the  time. 

When  comparing  the  percentage  agreement  results  on  the  first 
edition  of  the  ECC  and  the  revised  edition  of  the  ECC  with  the  fourth 
year  PT  students,  some  items  were  consistent  in  the  amount  of  agree¬ 
ment  shown.  Low  percentage  agreement  was  seen  on  both  instruments 
with  items  related  to  adhering  to  safety  precautions,  identifying 
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secondary  problems  and  recording  initial  evaluations.  The  conclusion 
drawn,  once  again,  is  that  preceptors  tend  to  judge  items  of  high 
importance  inconsistently.  Difficulty  in  discerning  the  extent  to 
which  safety  precautions  were  implemented  in  treatment  is  particularly 
disconcerting.  By  contrast,  high  percentage  agreement  was  evident 
across  both  evaluations  for  four  items:  tidiness  of  treatment  area, 
respect  for  confidentiality,  oral  communication  and  education  of  the 
patient/family.  It  appears  that  preceptors  judge  more  consistently 
items  which  show  more  uniformity  in  the  way  in  which  they  are 
performed. 

Of  the  seven  major  competence  areas,  items  related  to  implemen¬ 
tation  of  treatment  area  showed  the  highest  median  Kappa  coefficients 
at  .51  and  .55  for  fourth  and  third  year  PT  students  respectively. 
Moreover,  the  mean  percentage  agreement  of  items  was  highest  for  those 
items  under  the  implementation  of  treatment  area.  Since  items  in  this 
major  competency  area  were  readily  observed  and  frequently  rated 
through  the  clinical  placement,  there  was  more  information  available 
to  preceptors  for  determining  the  final  ratings  on  treatment  items, 
thus  promoting  consistency.  Further  to  this,  technical  skills  seem 
less  suspectible  to  variations  in  a  student's  approach  and/or  to  the 
preceptor's  individual  point  of  view. 

The  second  preceptors  tended  to  give  both  third  and  fourth  year 
students  lower  mean  ratings  than  did  the  first  preceptors,  but  this 
difference  was  not  significant  except  for  a  few  items.  The  items 
which  showed  a  significant  difference  between  the  mean  ratings  awarded 
by  the  two  preceptors  were  items  19,  28,  31  ,  36  and  38  for  the  fourth 
year  students  and  items  26,  38  and  39  for  the  third  year  students. 
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This  significant  difference  arose  when  di sproportionately  more,  "Not 
Observed",  ratings  were  used  by  the  second  preceptor.  The  missing 
data,  procured  by  the  high  usage  of  the  not  observed  rating,  limited 
analysis  and  interpretation  of  the  results. 

The  items  which  were  consistently  not  observed  with  both  third 
and  fourth  year  PT  students  were  those  related  to  preparing  the 

patient  for  discharge  and  assisting  others  implement  aspects  of  the 
patient's  PT  treatment.  Even  though  these  subcompetencies  were 
considered  important,  they  were  not  readily  observable  with  students. 
The  possible  reasons  for  this  situation  are  many.  Subcompetencies 
which  required  unobtrusive  observation  during  spontaneous  student- 
patient  interaction  did  not  lend  themselves  to  scheduled  observations 
by  two  preceptors.  This  notion  is  supported  by  the  finding  that  the 
second  preceptors,  who  were  assigned  to  students  for  daily  observation 
and  evaluation  only,  used  the  not  observed  rating  more  frequently  than 
did  the  first  preceptors,  who  were  also  designated  as  the  students' 

supervisors.  The  first  preceptors  may  have  had  more  opportunity  to 
observe  unobtrusi vely .  Another  possible  explanation  might  be  that 
preceptors  perceived  certain  subcompetencies  to  be  their  responsibi¬ 
lity  and  shielded  students  from  experience  in  these  areas.  Or  some 
subcompetencies  may  be  rare  occurrences,  particul ari ly  for  students 
who  practice  only  half  days. 

When  the  4-point  scale  was  reduced  to  a  3-point  scale,  the 

percentage  agreement  of  items  improve  dramatically  particul ari ly  for 

those  items  related  to  professional  behavior.  The  disagreement  seen 
between  competent  (3)  and  highly  competent  (4)  rating  pairs  may  be  due 
to  either  a  lack  of  clarity  in  the  meaning  of  the  two  levels  of 
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competence  or  preceptors'  inability  to  make  fine  discriminations  of 
competence.  The  conclusion  made  is  that  the  ECC  reliably  differen¬ 
tiates  between  incompetence  (fail)  and  competence  (pass)  but  does  not 
reliably  differentiate  between  competence  (pass)  and  a  high  level  of 
competence  (honors). 

Reliability  of  the  Entire  Instrument 

When  the  items  were  pooled  to  obtain  subscale  scores  for  the 
seven  major  competency  areas  and  to  obtain  a  total  scale  score, 
agreement  among  preceptors,  expressed  as  intraclass  reliability  coef¬ 
ficients,  was  quite  good.  The  reliabiity  of  the  ECC  total  scale 
scores  was  .591  for  fourth  year  PT  students  and  .624  for  third  year  PT 
students.  A  reliability  of  .591  indicates  that  35  percent  of  the 
variance  in  performance  by  individual  students  was  accounted  for  by 
actual  differences  in  performance.  The  remaining  variance  in  perfor¬ 
mance  was  due  to  preceptor  error  and  variable  clinical  situations.  In 
comparison  to  similar  clinical  evaluations  in  medicine,  the  reliabi¬ 
lity  results  for  the  ECC  surpassed  most  evaluations.  Irby,  Evans  and 
Larson  (1978)  suggested  that  reliability  of  clinical  evaluation  rarely 
exceeds  .30. 

The  improved  reliability  for  the  scale  scores  suggests  further 
that  the  ECC  functions  better  as  a  summative  evaluation  which  yields  a 
total  score  for  each  student  as  opposed  to  a  formative  evaluation 
which  provides  feedback  to  the  students  about  their  performance  on 
each  subcompetency.  The  total  scores  give  a  reliable  overall  assess¬ 
ment  of  the  competence  of  students. 

An  explanation  concerning  the  gradually  declining  reliability 
coefficients  over  the  three  testing  occasions  from  November,  1982  to 
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March,  1983  -  .704,  .624  and  .591  -  is  needed.  One  environmental 
factor  which  changed  during  the  course  of  the  study  was  an  increased 
work  demand  placed  upon  preceptors  due  to  economic  constraints  imposed 
by  hospital  administrations.  As  a  consequence,  preceptors  had  less 
time  available  to  devote  to  the  study.  Preceptors'  enthusiasm  toward 
the  study  may  have  also  waned  over  its  five  month  duration,  especially 
those  who  were  involved  in  two  or  more  of  the  testing  occasions. 

The  reliability  of  the  ECC  for  third  year  PT  students  at  .624 
was  good,  even  though  it  was  not  designed  for  their  use.  This  result 
is  likely  because  the  novice  students  performed  at  lower  levels  of 
competence  than  fourth  year  students,  where  discriminations  were 
easier  for  preceptors. 

In  relation  to  the  seven  subscale  scores  for  each  of  the  major 
competence  areas,  the  professional  behavior  area  had  the  lowest  intra¬ 
class  reliability  coefficients  for  both  the  fourth  year  and  third  year 
PT  students  at  .092  and  .257,  respectively.  The  professional  behavior 
area  was  the  section  of  the  ECC  designed  as  a  subjecti vely-worded 
rating  scale.  The  low  reliability  found  for  professional  behavior 
illusrates  that  a  behavioral ly-anchored  rating  scale  yields  better 
reliability  than  a  subjectively-worded  rating  scale. 

Val idi ty 

Content  validity  of  the  ECC  was  safeguarded  by  careful  planning 
in  the  development  phase  of  the  instrument.  Logical  analysis  revealed 
that  subcompetencies  included  in  the  ECC  were  representative  of  the 
behaviors  considered  necessary  for  acceptable  clinical  performance  in 
PT. 
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Concurrent  validity  of  the  ECC  was  found  by  comparison  with  the 
criterion  of  potential  employabi 1  i ty.  The  high  correlation  coeffi¬ 
cients  between  the  two  evaluations  suggested  that  ECC  scores  estimated 
potential  employabi lity.  The  hiring  rating  showed  a  high  reliability 
between  the  two  preceptors  which  gives  added  credence  to  its  use  as  a 
criterion  measure.  Caution  is  advised,  however,  in  the  interpretation 
of  the  relatively  high  correlation  between  these  two  evaluations  due 
to  the  possibility  of  criterion  contamination,  since  preceptors  were 
aware  of  how  the  student  performed  on  both  evaluations. 

Predictive  validity  was  not  investigated  in  the  present  study. 
The  main  purpose  of  the  study  was  to  evaluate  current  clinical 
competence  of  students,  not  to  predict  future  clinical  competence.  A 
follow-up  study  should  be  conducted  to  determine  if  ECC  scores  predict 
future  clinical  competence  in  a  job. 

Construct  validity  of  the  ECC  was  supported  by  confirmation  of 
the  hypothesis  that  experienced  fourth  year  PT  students  would  perform 
significantly  better  on  the  ECC  than  novice  third  year  PT  students. 
This  finding  clarifies  the  nature  of  clinical  competence  and  verifies 
that  the  ECC  actually  measures  clinical  competence.  The  ECC, 
therefore,  determines  the  extent  to  which  clinical  competence  has  been 
attained  by  PT  students. 

Usability 

The  primary  advantage  of  the  ECC  for  assessing  the  students' 
observed  performances  was  its  suitability  to  the  clinical  setting 
characterized  by  a  wide  range  of  patients  and  situations.  The  sub¬ 
competencies  in  the  ECC  were  general  in  nature  emphasizing  how  well 
the  student  integrated  and  applied  the  knowledge,  skills,  judgements 
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and  attitudes  for  effective  performance  and  optimal  care  to  patients. 
Consequently,  students'  behaviors  in  vastly  different  clinical 
settings,  such  as  a  burn  unit,  pediatric  unit,  intensive  care  unit  and 
home  care  service,  could  be  readily  translated  to  fit  the  ECC  format. 

The  reactions  of  the  preceptors  and  students  to  the  content, 
administration  and  effects  of  the  ECC  were  extremely  positive,  even 
though  it  was  a  longer  and  more  detailed  evaluation  than  ever  before. 
Communication  of  the  competencies  and  standards  helped  both  the 
preceptors  and  students  in  knowing  what  was  expected  and  to  what 
standard  of  performance.  The  daily  record  of  performance  was  also 
viewed  favorably  indicating  that  feedback  to  the  students  regarding 
their  performance  would  be  more  immediate  and  precise  as  well  as  being 
helpful  in  determining  a  final  rating  based  on  habitual  performance. 

The  ECC  could  be  administered  easily  in  any  physical  therapy 
clinical  program  requiring  only  minimal  instruction  in  its  use.  If 
utilized  for  grading  purposes,  the  ECC  is  reliable  for  determining  an 
incompetent  or  failing  grade  but  it  is  not  reliable  for  determining  an 
honors  level  of  competence. 
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Chapter  VI 


CONCLUSIONS  AND  RECOMMENDATIONS 

The  major  contributions  of  the  present  study  were  the  following: 

1.  Identification  of  clinical  competencies  in  order  of  importance 

to  the  practice  of  PT  helped  to  define  the  construct  of  clinical 
competence. 

2.  The  process  involved  in  developing  a  clinical  evaluation 

instrument  was  described. 

3.  A  cri terion-reference  evaluation  was  developed  which  fulfills  a 
need  in  PT  education  for  an  assessment  of  clinical  competence  of 
PT  students.  The  Evaluation  of  Clinical  Competence  was  deve¬ 
loped  to  determine  the  extent  to  which  a  PT  student  in  the  final 
year  of  the  training  program  has  achieved  the  clinical  com¬ 
petencies  necessary  to  be  eligible  for  admittance  into  the  PT 
profession. 

4.  Methods  suggested  in  the  measurement  literature  to  improve 

reliablity  and  validity  of  observational  evaluations  were 
incorporated  into  the  design  of  the  evaluation  in  an  attempt  to 
diminish  the  gap  between  what  is  theoretical ly  desirable  in 

assessment  of  students  and  what  evaluation  procedures  are 
actually  used  in  the  clinical  setting.  To  this  end,  a 
behavioral ly-anchored,  4-point  rating  scale  was  devised  to  judge 
the  quality  of  a  student's  performance  according  to  four 
standards  of  performance  expected  for  entry  into  the  PT 
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profession.  The  standards  of  performance  referred  to  what 
behaviors  a  student  must  display  to  warrant  a  certain  rating.  A 
daily  record  of  a  student's  performance  was  included  in  the 
evaluation  to  ensure  that  the  final  ratings  relied  on  the  actual 
behaviors  demonstrated  by  the  student  over  a  number  of  occasions 
with  different  patients  and  thereby  lessen  the  tendency  for 
preceptors  to  be  influenced  by  memory,  biases,  recency  effects 
and  an  atypical  performance  on  one  occasion  or  with  one 
difficult  patient. 

5.  The  reliability  and  validity  of  the  rating  scale  were  esta¬ 
blished  with  PT  students  from  the  UA. 

Recomnend ations 

The  ECC  with  its  known  reliability  and  validity  will  be  helpful 
to  both  educators  and  preceptors  in  making  major  decisions  regarding  a 
student's  clinical  competence,  promotion  and  graduation.  This 
competency-based  evaluation  offers  reasonable  precision  and  reliabi¬ 
lity  in  making  judgements  about  the  clinical  competencies  possessed 
and  applied  by  the  PT  student.  The  ECC  discriminates  between  incom¬ 
petence  or  a  failing  grade  and  competence  or  a  passing  grade;  however, 
caution  is  advised  in  the  use  of  the  ECC  to  discriminate  between 
competence  and  a  honors  level  of  competence.  The  ECC  will  not  only  be 
useful  in  student  education,  but  may  also  be  helpful  in  clinical 
certification  of  foreign-trained  physical  therapists  and  re-entry 
physical  therapists.  The  ECC  will  continue  to  evolve  with  further 
use,  validation  studies  and  feedback  from  students  and  preceptors. 


. 


v 


99 


Continual  testing  not  only  corrects  errors  of  measurement  but  assists 
users  to  define  more  clearly  what  they  are  measuring. 

The  ECC  requires  more  extensive  research  with  a  larger  sample  of 
PT  students.  Field  testing  with  PT  students  from  several  Canadian 
universities  during  their  summer  placements  may  be  feasible.  The 
extent  to  which  the  ECC  can  predict  future  job  performance  should  also 
be  investigated.  A  further  study  might  also  use  videotaped  recordings 
of  student  performance  on  selected  competence  areas  to  determine 
intra-rater  reliability  over  time.  The  training  of  preceptors  in  the 
use  of  the  ECC  may  also  be  studied  to  determine  the  effect  on  the 
rel i ability. 
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Competencies  to  be  Achieved  by  Physical  Therapy  Students 

Upon  Completion  of  Their  Clinical  Education  Program 


Upon  completion  of  a  clinial  education  program  in  PT,  the 
students  must  be  able  to  fulfill  the  following  competencies: 


A.  patient  evaluation 


The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  identify  and  gather  pertinent  patient  care 

information  (from  the  patient,  significant  others,  medical 
records  and  direct  physical  therapy  tests  and  evaluation 
procedures)  in  order  to  evaluate  the  patient's  neuro-musculo- 
skeletal  status,  and  to  evaluate  the  results  of  the  physical 
therapy  treatment  provided  by  the  student. 

The  student  will  demonstrate  competency  in  patient  evaluation  by 
meeting  the  following  cri teri a: 

1.  accurately  and  objectively  gathers  pertinent  patient 
information  from  medical  records  and  significant  others 
prior  to  the  initial  evaluation 

2.  accurately  and  objectively  gathers  pertinent  information 
from  the  patient  interview 

3.  selects  physical  therapy  (PT)  evaluation  procedures 
(includes  observation,  palpation  and  measurement) 
appropriate  to  the  patient's  disability,  age  and 
personal i ty 

4.  performs  selected  PT  evaluation  procedures  according  to 
standardized  or  recommended  technique 

5.  performs  selected  PT  evaluation  procedures  safely, 
thoroughly  and  efficiently 

6.  assesses  the  appropriateness  of  patient's  appliances/ 


equipment 

equipment 

and/or  assesses  the 

need 

for  appliances/ 

7. 

uti 1 i zes 

knowledge  of  normal 

human 

structures  and 

function,  pathological  conditions,  and  evaluation 
procedures  in  order  to  gather  data  relevant  to  the  neuro- 
musculo-skeletal  status  of  the  patient 

8.  explains  aspects  of  evaluation  procedures  and  findings 
that  should  be  explained  to  the  patient/family 

9.  recognizes  signs  of  the  patient's  apprehension  and 
responds  appropriately  to  ease  it 


. 
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10.  systematically  and  objectively  evaluates  the  results  of 
the  treatment  procedures  used 

11.  performs  a  comprehensive  re-evaluation  at  appropriate 
intervals  to  determine  effectiveness  of  treatment  outcome. 


B.  program  planning 

The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  interpret  and  integrate  evaluation  data  in  order 
to  identify  the  patient's  problems  and  capabilities,  and  to  plan 
a  progressive  physical  therapy  treatment  program  using 
procedures  which  are  consistent  with  the  patient's  status,  and 
physical  and  psychological  needs/goals. 

The  student  will  demonstrate  competency  in  program  planning  by 
meeting  the  following  criteria: 

1.  utilize  knowledge  of  pathology  and  normal  human  structures 
and  functions  to  interpret  and  integrate  evaluation  data 
gathered  from  all  sources 

2.  identifies  primary  problems  which  may  be  alleviated  by 
physical  therapy 

3.  identifies  secondary  problems  which  may  be  prevented  by 
physical  therapy  and  those  requiring  referral  to  other 
health  care  services 

4.  organizes  the  primary  problems  in  order  of  importance 

5.  identifies  the  patient's  capabilities 

6.  suggests  referral  to  other  health  care  services  when 
indicated 

7.  determines  short-term  goals  to  alleviate  primary  problems 

8.  establishes  long-term  goals  which  are  realistic  to  the 
probable  discharge  situation. 

9.  selects  appropriate  PT  treatment  procedures  which  are 
consistent  with  short-  and  long-term  goals 

10.  presents  alternative  solutions  to  problems 

11.  justifies  his/her  choice  of  treatment  procedures  utilizing 
knowledge  of  PT  treatment  procedures  and  the  patient's 
condition 
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12.  informs  the  referring  physician,  when,  in  the  student's 
judgement,  a  change  in  the  requested  treatment  would  be 
more  beneficial 

13.  treatment  plan  demonstrates  progression  in  a  logical 
sequence  and  is  consistent  with  the  patient's  physical  and 
psychological  needs 

14.  selects  the  correct  frequency  and  duration  of  each 
treatment  procedure 

15.  treatment  plan  provides  for  re-assessment  of  the  outcome 
of  chosen  PT  treatment  procedures 

16.  includes  the  patient  in  the  planning  process  by  guiding 
the  patient's  determination  of  goals  and  selection  of  PT 
treatment  procedures 

17.  obtains  the  patient's  informed  consent  prior  to 
commencement  of  treatment. 


C.  implementation  of  the  treatment  program 

The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  organize  and  effectively  perform  patient 
treatments  and  progressions,  within  the  recommended  standards  of 
quality  care  and  safety  for  both  the  patient  and  student,  in 
order  to  provide  optimal  patient  care. 

The  student  will  demonstrate  competency  in  implementation  of  the 
treatment  program  by  meeting  the  following  criteri a: 

1.  positions,  moves  and  drapes  the  patient  to  ensure  comfort, 
dignity  and  safety 

2.  implements  the  treatment  plan  safely  and  effectively 
utilizing  knowledge  of  PT  treatment  procedures 

3.  observes  the  patient  continually  to  assess  patient's 
clinical  signs  and  response  to  the  treatment  given 

4.  modifies  existing  treatment  procedures  to  obtain  the 
patient's  best  performance  and  interest 

5.  treats  the  patient  within  his/her  limits  of  disability, 
capability,  age,  tolerance  (fatique,  pain) 

6.  implements  or  suggests  new  treatment  procedures  or 
progressions  consistent  with  observations  and  ongoing 
evaluations 


. 
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7.  recommends  appropriate  time  of  discharge  from  PT  services 

8.  prepares  patient/family  for  discharge  in  adequate  time 

9.  develops  and  records  the  post-discharge  program  for  the 
patient/family  keeping  comnunity  and  hospital  resources  in 
mind 

10.  prepares  treatment  area  prior  to  treatment 

11.  cleans  up  treatment  area  and  equipment  after  use 

12.  adheres  to  all  precautions  associated  with  equipment, 
modalities  and  treatment  procedures  (e.g.  checks  equipment 
prior  to  use) 

13.  uses  correct  body  mechanics  when  applying  techniques  and 
when  moving  patients  and/or  equipment 

14.  knows  hospital  emergency  procedures  and  sterile  technique 

15.  reports  need  for  equipment  repair  and  need  of  supplies. 


C.  coirmuni cation  with  patient/family 

The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  establish  communication  at  a  level  appropriate  for 
the  patient/family  in  order  to  develop  a  therapeutic 
relationship  conducive  to  their  understanding,  motivation  and 
confidence  regarding  the  therapeutic  process  and  the  patient's 
condition. 

The  student  will  demonstrate  competency  in  communication  with 
patient/family  by  meeting  the  following  cri teri a: 

1.  educates  the  patient/family  regarding  the  patient's  status 
and  treatment  procedures  as  well  as  their  importance  or 
impl ications 

2.  adjusts  approach  and  level  of  conmunication  to  the 
patient's/family's  age,  education  and  understanding 

3.  exhibits  'caring'  attitude  toward  the  patient/family  by 
listening  to  them  express  their  attitudes  and  feelings 

4.  informs  or  prepares  the  patient/family  for  progressions  or 
changes  in  treatment  provision 

5.  gives  the  patient/family  clear  and  concise  instruction 
during  treatment 
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6.  uses  praise  and  other  reinforcers  to  encourage  the 

patient/fami 1 y 

7.  relieves  the  patient's/family's  apprehension  and  instills 

confidence 

8.  sets  appropriate  limits  on  the  patient's  behavior  that 

interferes  with  treatment 

9.  teaches  the  patient/family/other  appropriate  health  care 

personnel  ways  in  which  to  implement  or  continue  treatment. 


E .  communication  with  health  care  personnel 

The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  effectively  give  and  receive  patient  care 

information  by  verbal  and  nonverbal  methods  in  order  to 

establish  professional  relations  as  a  member  of  the  health  care 
delivery  system. 


The  student  will  demonstrate  competency  in  communication  with 
health  care  personnel  by  meeting  the  following  criteri a: 

1.  communicates,  in  oral  or  written  form,  results  of 

evaluation  to  the  physician,  supervising  therapist  and 
other  appropriate  health  care  personnel 

2.  communicates,  in  oral  or  written  form,  observations, 
results  of  treatment  and  treatment  progressions  or  changes 
to  the  appropriate  personnel 

3.  presents  oral  reports  in  form,  content  and  manner 
appropriate  for  the  intended  audience  (e.g.  patient 
conferences,  rounds) 


4.  correctly  interprets  and 

communication  from  others 

5.  exhibits  perceptiveness, 

communication  with  others 

6.  respects  confidentiality  of 

7.  interprets  the  roles  and 

according  to  their  level  of 


utilizes  oral  and  written 

tact  and  discretion  in 

information 

functions  of  PT  to  others 
interest  and  understanding. 


F.  documentation 


The  student  must  demonstrate  the  knowledge  and  skills  with  which 
to  document  patient  care  information  in  medical  and  statistical 


. 
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records  in  accordance  with  the  policies  of  the  facility  and 
professional  clinical  practice. 

The  student  will  demonstrate  competency  in  documentation  by 
meeting  the  following  cri teri a: 

1.  documents  evaluation  results  in  the  patient  care  record  in 
a  manner  which  is  objective,  pertinent,  accurate, 
thorough,  concise  and  legible 

2.  documents  problem  lists,  treatment  goals  and  treatment  plan 

3.  .documents  observations,  treatment  procedure  modifications 

or  progressions  which  are  consistent  with  observations  and 
ongoing  evaluation 

4.  documents  patient  data  according  to  the  format  of  the 

facility  and  within  specified  time  limits 

5.  documents  data  using  accurate  medical  terminology 

6.  uses  Canadian  System  of  Unit  Values  to  accurately  record 
statistical  data. 


G.  personal/professional  behavior 

The  student  must  demonstrate  personal/professional  behavior  by 
adherence  to  legal,  ethical,  professional  and  facility 
requirements . 

The  student  will  demonstrate  competency  in  personal /professional 
behavior  by  meeting  the  following  cri teri a: 

1.  presents  a  professional  appearance 

2.  accepts  responsibility 

3.  assists  and  cooperates  willingly  with  co-workers 

4.  accepts  constructive  criticism  and  suggestions,  and 
modifies  behavior  accordingly 

5.  handles  personal  and  professional  frustration  appropri ately 

6.  is  reliable  regarding  punctuality  and  duties. 


H.  personal/professional  growth 

The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  critically  examine  his/her  personal/professional 
performance  and  to  pursue  educational  and  research  activities 
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which  promote  personal/professional  growth  and  continuing 

competency. 

The  student  will  demonstrate  competency  in  personal/professional 
growth  by  meeting  the  following  criteri a: 

1.  identifies  own  strengths  and  weaknesses 

2.  asks  appropriate  questions  or  seeks  assistance  when  in 
doubt 

3.  uses  added  knowledge  and  self-assessment  to  improve 
.overall  performance 

4.  seeks  out  available  resources,  references  and  inservice 
education  to  enhance  knowledge  and  skills  and  to  keep  up 
with  new  developments 

5.  uses  unscheduled  time  in  an  effective  and  beneficial  manner 

6.  demonstrates  problem-solving  skills  in  patient  management 

7.  interprets  and  utilizes  results  of  research  performed  by 
others. 


I .  management  of  direct  patient  care 

The  student  must  demonstrate  the  knowledge,  skills  and  attitudes 
with  which  to  facilitate  the  provision  of  comprehensive  patient 
care. 

The  student  will  demonstrate  competency  in  management  of  direct 
patient  care  by  meeting  the  following  criteri  a: 


1. 

organize  case-load  and  coordinates  treatment 
patient/PT  staff/other  therapies 

times 

wi  th 

2. 

schedules  self  the  appropriate  amount  of  time 
preparation,  completion  and  documentation  of  treatment 

for 

3. 

adjusts  schedule  to  meet  unforeseen  events 

4. 

supervises  support  PT  personnel  appropri ately 

5. 

follows  or  recommends  alternative  operational 
procedures  related  to  direct  patient  care 

pol icies 

and 

6. 

assumes  the  PT  role  or  function  (prevention, 
consultation,  community-based)  appropriate  to 
care  setting 

maintenance, 
the  health 

. 
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aware  of  functions  of  other  disciplines  involved  in  the 
patient  care  team  and  collaborates  with  these  disciplines 
to  provide  comprehensive  patient  care. 


APPENDIX  B 

Questionnaire  to  Rate  Importance  of  Competencies  in 
Physical  Therapy  and  Accompanying  Letter 
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Dear 

I  am  writing  to  ask  for  your  assistance  in  a  project  in 
which  I  am  developing  a  clinical  evaluation  instrument, 
Evaluation  of  Clinical  Competence.  The  purpose  of  this 
evaluation  is  to  determine  whether  a  fourth  year  physical 
therapy  student  has  achieved  the  competencies  which  enable 
him/her  to  perform  adequately  as  a  new  practitioner  following 
graduation.  As  part  of  the  instrument  development  phase, 

I  need  assistance  from  physical  therapists  in  clinical 
practice  to  rate  the  degree  of  importance  of  certain 
competencies  to  provision  of  patient  care.  The  rationale 
for  this  project  is  explained  more  fully  in  the  instructions 
which  accompany  the  competencies  to  be  rated. 

I  have  enclosed  a  number  of  competency  rating  forms  v/ith 
an  attached  instruction  page  to  facilitate  correct 
completion  of  the  rating  form.  I  would  appreciate  if  as 
many  therapists  in  your  department  as  are  willing  would 
participate  in  the  rating.  The  ratings  are  extremely 
important  as  to  the  score  a  student  will  receive  on 
a  competency  in  this  newj evaluation  form. 

To  protect  the  therapist's  confidentiality,  would  you 
please  distribute  the  competency  rating  forms  to  the  ther¬ 
apists  in  your  department.  I  have  asked  the  therapists  to 
return  the  completed  rating  forms  to  you  within  a  two  week 
period.  Then,  if  you  would  please  return  these  forms  to  me 
in  the  stamped  and  self-addressed  envelope  provided, 
the  identity  of  the  hospital  would  also  be  protected. 

Thank  you  very  much  for  any  assistance  that  you  and 
the  therapists  in  your  department  can  give  me. 


Sincerely, 


An-4 


Joan  Loomis 

Department  of  Physical  Therapy 
210  Corbett  Hall 
University  of  Alberta 
Edmonton,  Alberta  T6G  2G4 
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Competency  Rating  Form 


Preamble 

I  am  currently  developing  a  clinical  evaluation  instrument 
to  determine  if  fourth  year  physical  therapy  students  have 
achieved  the  competencies  required  for  adequate  performance 
upon  entering  the  physical  therapy  profession. 

The  first  step  in  the  development  of  an  instrument 
designed  to  evaluate  clinical  competence  is  to  identify 
clinically  significant  competencies.  A  number  of  competencies 
have  been  identified  from  a  survey  of  the  literature  and 
professional  documents. 

The  second  step  in  the  instrument  development  phase  is  to 
rate  each  competency  according  to  its  degree  of  importance 
in  the  provision  of  direct  patient  care.  This  rating  will  be 
used  to  weigh  each  competency  on  the  evaluation  form  so  that 
its  degree  of  importance  influences  the  score  awarded 
the  student. 

You  would  assist  me  greatly  in  the  development  of  this 
instrument,  Evaluation  of  Clinical  Competence,  if  you  would 
rate  each  competency  according  to  how  important  you  feel 
that  it  is  to  clinical  practice  in  pysical  therapy. 

Importance  ratings  are  influenced  by  such  factors  as 
frequency  of  occurrence,  life-threatening  or  harmful  aspects 
of  inadequate  performance  or  values  espoused  by  the  physical 
therapy  profession. 

Instructions 

1.  Flease  circle  on  the  accompanying  pages  the  rating  that 
you  feel  most  indicates  how  important  each  competency  is 
to  clinical  practice  in  physical  therapy: 


nonessential 
minor  importance 
important 
very  important 
essential 


noness . 
minor 
imp. 
v .  imp . 
ess . 


2.  Please  do  not  write  your  name  on  any  of  the  pages  to 
ensure  confidentiality. 

3.  Please  return  the  completed  competency  rating  form  to 
your  department  supervisor  within  a  two  week  period.  He/ 
she  will  return  them  all  to  me  in  an  unmarked  self- 
addressed  envelope  to  safeguard  against  identification  of 
yourself  and  your  employer. 

(4.  If  you  are  interested,  you  may  wish  to  reword  any 

competencies  for  clarity  or  suggest  any  competencies 
that  may  have  been  overlooked). 


Joan  Loomis 


' 
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lircla  the  rating  that  you  feel  most  indicates  how  important 
each  competency  is  to  clinical  practice  in  physical  therapy. 


Key:  noness  -  r.onessential 

minor  -  minor  importance 
imp  -  importance 
v.imp  -  very  important 
ess  -  essential 


w once tenc res 


patient  evaluation 


noness 

minor 

imp 

v.imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

f 

noness 

minor 

1 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v.imp 

ess 

noness 

minor 

imp 

v.imp 

t 

ess 

none ss 

minor 

t 

-imp 

v .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.imp 

ess 

noness 

minor 

imp 

v .  imp 

1 

ess 

noness 

minor 

imp 

v.  imp 

ess 

» 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.imp 

ess 

noness 

i 

minor 

imp 

v.imp 

» 

ass 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v.imp 

ess 

♦ 

noness 

minor 

imp 

v .  imp 

ess 

1.  accurately  and  objectively  gathers  pertinent  patient  information 
from  medical  records  and  significant  others  prior  to  the  evaluation 

2.  accurately  and  objectively  gathers  pertinent  information  from 
the  patient  interview 

3.  selects  ?T  evaluation  procedures  (includes  observation,  palpation 
and  measurement)  appropriate  to  the  patient's  disability,  age 
and  personality 

4.  performs  selected  ?T  evaluation  procedures  correctly,  safely, 
accurately  and  proficiently 

5.  assesses  the  appropriateness  of  the  patient's  appliances/equipment 
and/or  assesses  the  need  for  appliances/equipment 

6.  utilizes  icnowledge  of  anatomy,  physiology,  pathological  conditions 
and  evaluation  procedures  in  order  to  gather  data  relevant  to 

the  patient's  3tatus 

7.  explains  aspects  of  the  evaluation  orocaduras  and  findings  that 
should  be  explained  to  the  patier.t/famiiy 

3.  recognizes  signs  of  the  patient's  apprehension  during  the 
evaluation  and  responds  appropriately  to  ease  it 

9.  performs  a  comprehensive  re-evaluation  at  appropriate  intervals 
to  determine  effectiveness  of  the  treatment  outcome 

Competencies  in  program  Planning 

I. 0.  utilizes  knowledge  of  pathology,  anatomy  and  physiology  to 

interpret  and  integrate  evaluation  data  gataered  from  all  sources 

II.  identifies  and  prioritizes  primary  problems  which  may  be 
alleviated  by  physical  therapy 

12.  identifies  and  prioritizes  secondary  problems  which  may  be 
prevented  by  ?T  and  those  requiring  referral  to  other  health 
care  services 

13.  determines  short-term  goals  to  alleviate  primary  problems 

14.  determines  long-term  goals  which  are  realistic  to  the  probable 
discharge  situation 

15.  selects  ?T  treatment  procedures  (includes  selection  of  sequence, 
frequency,  duration  and  progression  of  procedures)  appropriate 
to  the  patient's  physical  ar.d  psychological  goals  and  needs 

lo.  utilizes  'knowledge  of  the  physiological  effects  of  ?T  treatment 
and  the  pathology  of  the  patient’s  condition  to  defend  his/her 
choice  of  treatment  procedures  with  physicians  and  supervising 
therapist 

£•17.  includes  the  patient/family  appropriately  in  the  planning  process 
by  guiding  the  patient ' s/family ' s  determination  of  goals  and 
selection  of  ?T  treatment  procedures,  and  obtains  the  patient's 
informed  consent  prior  to  commencement  of  treatment 


Key: 


noness  -  nonessential 
tiinor  -  minor  importance 
imp  ~  importance 
v.imp  -  very  important 
es3  -  essential 


noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

s 

v.imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v.imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

t 

. 

noness 

r 

minor 

imp 

v .  imp 

ess 

noness 

f 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

V .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

1 

t 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

e 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

1 

noness 

minor 

imp 

v .  imp 

ess 

i 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

Competencies  in  implementation  of  the  treatment  program 

13.  positions,  u.oves  and  drapes  the  patient  to  ensure  comfort, 
dignity  and  safety 

19-  implements  the  treatment  plan  in  a  correct,  safe,  effective  and 
proficient  manner 

20.  observes  the  patient  continually  to  assess  patient's  clinical 
3igns  and  response  to  the  treatment  with  respect  to  the  patient's 
disa'oility/age/toleranoe 

21.  modifies  or  progresses  or  implements  new  treatment  procedures 
.in  accordance  with  student's  observations  and  ongoing  evaluations 

and  in  order  to  obtain  the  patient's  best  performance  and  interest 

22.  accepts  responsibility  for  preparation,  care  and  tidiness  of 
the  equipment  and  treatment  area,  and  reports  need  for  supplies 
ar.d  equipment  repair 

23*  schedules  self  and  organises  patient  schedule  in  collaboration 
with  the  patient/PT  staff/other  disciplines/unforsseen  events 

24..  adheres  to  all  safety  precautions  associated  with  equipment, 
modalities  and  treatment  procedures 

25.  uses  correct  body  mechanics  when  applying  techniques  and  when 
moving  patient  and/or  equipment 

26.  recommends  and  prepares  patient/family  for  discharge  in  adecuate 
time  and  develops  a  suitable  post-discharge  program  -ceeping" 
community  and  hospital  resources  in  mind 

Competencies  in  communication  with  oatisnt/family 

27.  educates  the  patient/family  regarding  the  patient's  status  and 
?T  treatment  procedures,  progressions  and  discharge  plans 

23.  adjusts  approach  and  level  of  communication  to  the  patient's/ 
family's  age,  education,  comprehension,  apprehension  and  confider.c 

29.  gives  clear  and  concise  instructions/demonstrations  to  motivate 
and  encourage  the  patient/family  during  treatment 

30.  uses  praise  and  other  feedback  to  encourage  the  patient/family 

31.  exhibits  'caring'  attitude  toward  the  patient/family  by  listening 
to  them  express  their  attitudes  and  feelings  and  by  setting 
appropriate  limits  on  behavior  that  interferes  with  treatment 

32.  teaches  the  patient/family  ways  in  which  to  implement  or  continue 
*  r  2  3.  "tin  0  n  t 

Competencies  in  communication  with  'Other  health  care  personnel 

33 •  communicates  patient  care  information  orally  to  the  physician/ 
supervising  therapist/appropriate  others  in  form,  content  and 
manner  appropriate  for  the  intended  audience 

34.  correctly  interprets  and  utilizes  oral  and  written  communication 
from  others 

35*  exhibits  perceptiveness,  tact  and  discretion  in  communication 
with  others 

36.  respects  confidentiality  of  information 

37.  assists  other  health  care  personnel  implement  aspects  of  the 
patient's  ?T  treatment  according  to  their  level  of  interest 
and  understanding 


A 


_ 
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Key:  noness  -  nonessential 

minor  -  minor  importance 
imp  -  importance 
v. imp  -  very  important 
ess  -  essential 


Competencies 


iocunencarion 


noness 

nj.no  r 

imp 

v.  imp 

es3 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

t 

imp 

v .  imp 

ess 

noness 

minor 

imp 

'/.  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

mino  r 

imp 

•/.  imp 

t 

ess 

noness 

minor 

imp 

•/.imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

r.  imp 

ess 

1 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

/ 

noness 

minor 

imp 

*/ .  imp 

ess 

38.  documents  evaluation  results  in  the  patient  care 
a  manner  which  is  relevant,  accurate,  objective, 
concise  and  legible 


record  in 
complete , 


39.  documents  complete  and  appropriate  problem  list,  treatment 
plan,  progression  of  the  treatment  plan  and  patient  progress 

40.  documents  patient  records  and  statistical  records  according 
to  the  format  and  time  limits  of  the  facility 

Competencies  in  oersor.al/orofessional  behavior 

41 .  presents  a  professional  appearance 

42.  accepts  responsibility 

43.  assists  and  cooperates  willingly  -with  co-workers 

44.  accepts  suggestions  and  modifies  behavior  accordingly 

45.  handles  personal  and  professional  frustrations  appropriately 

46.  is  reliable  regarding  punctuality  and  duties 


Competencies  in  oersonal/orofessional  growth 

47.  identifies  own  strengths  and  weaknesses 

48.  asks  appropriate  questions  or  seeks  assistance  when  in  doubt 

49.  uses  added  knowledge  and  self-assessment  to  improve  overall 
performance 

50.  seetfs  out  available  resources,  references  and  inservice  education 

to  enhance  knowledge  and  skills  and  to  keep  up  with  new  developments 

51.  interprets  and  utilizes  results  of  research  performed  by  others 
Competencies  in  management  related  to  direct  patient  oare 

32.  supervises  support  FT  personnel  appropriately 

53-  follows  or  recommends  alternative  operational  procedures  related 
to  provision  of  patient  care 

54.  assumes  the  FT  role  (prevention,  maintenance,  consultation, 
community-based)  appropriate  to  the  health  car.e  setting 

53-  is  aware  of  functions  of  other  disciplines  involved  in  the  patient 
care  team  and  collaborates  with  these  disciplines  to  provide 
comprehensive  patient  oare 


- 
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Key:  noness  -  nonessential 

minor  -  minor  importance 
imp  -  importance 
v.imp  -  very  important 
ess  -  essential 


noness 

nu.no  r 

-its? 

v .  imp 

ass 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.  lap 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v.  imp 

ess 

noness 

minor 

imp 

v.  iap 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

v .  imp 

ess 

noness 

minor 

imp 

/ .  imp 

ess 

Since  the  whole  is  not  necessarily  the  sun  o?  its  parts,  you 
are  now  asked  to  rats  the  importance  of  the  global  competencies 
which  encompass  the  detailed  competencies  that  you  have  just 
finished  rating.  Please  circle  the  rating  that  you  feel  most 
indicates  how  important  each  global  competency  is  in  physical 
therapy  practice. 

1 .  patient  evaluation 

2.  program  planning 

3-  implementation  of  the  treatment  program 

1.  communication  with  the  patient/family 

5.  communication  with  other  health  care  personnel 

6.  documentation 

7.  personal/professional  behavior 

3.  personal/professional  growth 

9.  management  related  to  direct  patient  care 


Please  indicate  in  which  specialty 

.general  _ cardiopulmonary 

orthopaedics  _ rheumatology 


area  or  areas  that  you  work. 

_ private  practice/outpatien 

_ sports  therapy 


neurology 


bums 


. 
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DIRECTIVE 

Evaluation  of  Clinical  Competence  Form  (ECC):  Fourth  Year  PT  Students 


This  instrument  is  designed  to  evaluate  clinical  competence  as 
demonstrated  by  fourth  year  PT  students  throughout  a  placement 
according  to  four  standards. 

The  instrument  contains  two  sections: 

Section  I:  Daily  Ratings  of  Patient  Care  Competencies 

Students  are  rated  daily  on  their  direct  patient  care  skills  in 
order  to  determine  by  the  end  of  the  placement  a  rating  of  clinical 
competence,  which  reflects  their  performance  throughout  the  duration  of 
the  placement. 

Section  I  contains  six  major  competency  areas:  Patient 
evaluation,  Program  planning.  Implementation  of  treatment, 

Communication  with  patient/family,  Communication  with  other  health  care 
personnel  and  Documentation. 

Section  II:  Rating  of  Professional  Competencies 

The  purpose  of  this  rating  is  to  determine  the  frequency  with 
which  professional  characteristics  are  demonstrated  by  the  student  in 
the  final  week  of  placement. 

Section  II  contains  one  major  competency  areas:  Professional 
behavior  and  growth. 


Instructions  for  Daily  Ratings:  Section  I  of  ECC  Form 


A.  General  Instructions 


1 .  Two  raters  observe  and  rate  the  student  once  daily  throughout  the 

placement .  The  first  rater  is  the  student's  supervising 
therapist.  The  second  rater  is  assigned  to  the  student  for  daily 
observations  and  ratings  only.  The  student  is  introduced  to  the 
second  rater  on  DAY  1  of  the  placement. 

2.  Daily  ratings  commence  on  DAY  4  of  the  placement  and  terminate 
after  DAY  17  of  the  placement.  A  total  of  14  daily  ratings  is 
possible  if  the  student  and/or  raters  are  not  absent. 

B.  Observation  Instructions 

1.  By  DAY  4  the  student  will  prepare  a  list  of  patients  that  he/she 
would  like  to  treat  and/or  discuss  in  the  presence  of  either 
rater.  This  list  is  updated  by  the  student  periodically  on  the 
request  of  the  first  rater  or  on  the  student's  own  volition. 


. 
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The  two  raters  meet  at  the  beginning  of  each  day  from  DAY  4  to  DAY 

17  of  the  placement  to  decide: 

a.  which  of  the  student’s  patients  from  the  list  will  be  used 
for  observation  on  that  day 

b.  which  method  of  observation  will  be  used  on  that  day. 

The  methods  of  observation  include  the  following: 

1.  a  patient  treatment  or  evaluation 

ii  .  a  team  meeting 

iii.  a  discussion  of  program  planning 

iv.  an  evaluation  of  the  student's  documentation  in  a  patient’s 
chart. 

Guidelines  for  determining  which  method  of  observation  is  to  occur 

each  day  are  the  following: 

a.  select  the  method  of  observation  which  is  most  likely  to 
occur  that  day 

b.  vary  selection  of  observation  from  day  to  day  so  that  each 
major  competency  area  is  observed  a  minimum  of  two  times  at 
different  intervals  throughout  the  placemment 

c.  vary  selection  of  patient  from  day  to  day  so  that  the  student 
is  observed  with  different  patients  over  the  placement. 

At  the  daily  meeting  from  DAY  4  to  DAY  17  the  two  raters  also 

determine  whether  observation  on  that  day  is  conducted  by  the 

raters  together  or  separately . 

Observation  by  Raters  Together*** 

a.  Observations  by  both  raters  together  is  the  preferred  and 
recommended  method. 

b.  The  two  raters  agree  upon  a  time  to  observe  the  student 
together.  The  student  is  then  informed  of  the  patient 
involved  and  the  observation  time.  The  onus  is  on  all 
parties  to  be  prepared  for  observation  at  this  pre-determined 
time . 

c.  Joint  observation  by  the  two  raters  is  essential  for 
observation  of  the  student  during:  i.  a  team  meeting 

ii .  a  discussion  of 
program  planning. 


Observation  by  Raters  Separately 

a.  When  joint  observation  by  raters  is  NOT  feasible  in  your 

setting,  the  two  raters  can  observe  the  student  separately 
particularly  with  regard  to: 
i  .  a  patient  evaluation 

ii .  a  patient  treatment 

iii.  an  evaluation  of  the  student's  documentation  in  a 
patient's  chart 
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b.  If  it  is  necessary  to  conduct  separate  observations,  each 
rater  makes  a  10-15  minute  appointment  with  the  student  to 

observe  him/her  with  any  one  of  the  patients  listed  by  the 
student.  The  onus  is  on  all  parties  to  be  prepared  for 
observation  at  the  pre-determined  times. 

7.  If  at  all  possible,  raters  observe  the  student  together  with  the 
same  patient.  However,  it  is  imperative  that  both  raters: 

a .  observe  the  student’s  performance  for  the  same  amount  of 

time ,  eg.  10-15  minutes  per  day 

b.  observe  performance  either  continuously  or  by  sampling  (eg. 
raters  sample  performance  by  observing  for  5  minutes  at  5 
minute  intervals  over  a  30  minute  evaluation  or  treatment 
session) 

c .  observe  and  rate  all  the  major  competency  areas  in  Section  I 

of  the  form  at  least  two  times. 

8.  Guidelines  for  lessening  the  student’s  anxiety  during  the 
scheduled  observations  are: 

a.  The  student  must  be  made  to  feel  as  comfortable  as  possible 
by  making  the  observations  informal  and  by  having  no 
evaluation  forms  in  view. 

b.  The  student  must  realize  that  observations  are  for  the 
purpose  of  evaluating  how  well  the  form  measures  clinical 

competence .  So  that  that  student  may  also  benefit  from  these 
observations,  both  the  first  and  second  raters  may  use  the 
observation  time  to  provide  the  student  with  a  positive 
learning  experience.  Both  raters,  therefore,  ask  or  respond 
to  questions,  make  corrections  or  suggestions  and  provide 
assistance,  as  necessary.  The  raters  also  provide  praise  and 
feedback  to  assist  the  student  to  attain  his/her  optimal 
level  of  performance.  (If  you  prefer  not  to  correct  the 
student  in  front  of  the  patient,  count  the  number  of  times 
that  you  would  have  made  corrections  in  order  to  determine  a 
rating) . 

c.  The  second  rater  only  interacts  with  the  student  during  these 
scheduled  observation  times.  Since  the  second  rater  does  not 
necessarily  know  the  patient,  caution  is  advised. 

d .  The  student  and  rater  understand  that  any  information  gained 

during  these  observations  will  not  be  utilized  to  write  up 

the  U  of  A  Evaluation  Form  and  will  not  unjustifiably 

influence  the  student's  grade  in  this  placement. 

C.  Rating  Instructions 

1.  Immediately  or  as  soon  as  possible  after  observing  the  student's 
performance,  the  two  raters  independently  rate  the  student's 
performance  on  the  subcompetencies  under  all  of  the  relevant  major 

competency  headings  in  Section  I  of  the  form. 

No  joint  discussions  or  joint  decisions  are  made  by  the  two  raters 

during  either  observation  or  rating  of  the  student's  performance. 


2. 
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3.  Using  Section  I  of  the  ECC  Form,  the  rater  independently  chooses 
that  standard  of  performance  (1,  2,  3,  4)  from  the  four  choices 
provided  under  each  subcompetency  which  best  describes  the 
student's  performance  on  the  subcompetency. 

4.  If  a  subcompetency  has  not  been  observed,  the  rater  chooses  that 
"not  observed”  option  (0) . 

5.  Adjacent  to  each  subcompetency,  the  rater  writes  in  the  number 
(0,1, 2, 3, 4)  representing  the  standard  of  performance  that  the 
rater  has  chosen.  This  number  is  written  in  the  column 
corresponding  to  the  day  of  the  evaluation. 

6.  When  choosing  a  standard  of  performance,  a  rater  must  base  his/her 
decision  on  the  definition  of  terms  provided  on  p.  1  of  the 
evaluation  form. 

7.  At  the  top  of  the  same  column  as  the  daily  rating,  circle  either 
the  't'  (together)  or  the  ’s'  (separate)  to  indicate  how  the 
observation  was  conducted  on  that  day. 


Instructions  for  Final  Ratings:  A.  Section  I  of  ECC  Form 

1 .  Final  ratings  of  Section  I  are  completed  along  with  Section  II 
according  to  your  form  completion  sequence  anytime  between  DAY  18 
and  DAY  20. 

2.  Between  DAY  18  and  DAY  20,  the  raters  independently  review  the 
student's  daily  ratings  and,  then,  independently  choose  a  rating 
which  best  describes  the  student's  overall  performance  in  each 

subcompetency .  Thi s  number  is  written  in  the  column  corresponding 
to  DAY  20.  Both  raters  must  attempt  to  give  an  overall  rating  of 
all  observed  subcompetencies  in  Section  I,  even  if  a  subcompetency 
was  observed  only  1-2  times. 


B.  Section  II  of  ECC  Form 

1.  See  page  7  of  the  evaluation  form. 

2.  Section  II  is  completed  along  with  Section  I  according  to  your 
form  completion  sequence  anytime  between  DAY  18  and  DAY  20. 


Thank  you  for  your  efforts! 


Joan  Loomis 
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The  University 
of  Alberta 


Department  of  Physical  Therapy 
Faculty  of  Rehabilitation  Medicine 


Edmonton,  Alberta,  Canada  T6G  2G4 
Telephone  (403)  432-5983 


Joan  Loomis  will  be  conducting  research,  in  conjunction  with 
the  Department  of  Physical  Therapy  at  the  University  Alberta, 
on  an  evaluation  instrument  that  she  developed  to  evaluate 
clinical  competence  of  physical  therapy  students.  The 
development  of  a  valid  and  reliable  evaluation  of  clinical 
competence  has  future  implications  for  students,  physical 
therapy  educators,  employers  of  new  graduates  and 
professional  growth.  It  is  hoped  that  your  physical  therapy 
department  will  agree  to  participate  in  this  much  needed  and 
worthwhile  study. 

Enclosed  are  two  informed  consent  forms:  1)  explanation  of 
field  testing  and  2)  consent  form.  The  explanation  of  field 
testing  outlines  the  rationale  of  the  study,  the  design  of 
the  evaluation  instrument,  and  the  method  of  administering 
this  evaluation  in  order  to  test  its  validity  and 
reliability.  Please  read  the  explanation  carefully  and 
contact  Joan  Loomis  (432-2071)  if  you  have  any  further 
questions.  If  you  are  able  to  assist  us  with  this  study, 
please  complete  the  consent  form  and  return  to  Joan  Loomis  at 
the  Department  of  Physical  Therapy,  University  of  Alberta. 
Furthermore,  if  you  would  appoint  a  liaison  person,  such  as  a 
clinical  instructor,  with  whom  Joan  may  communicate  regarding 
the  administration  of  this  evaluation  with  fourth  year 
students  in  January  and  March,  1983  and  with  third  year 
students  in  February,  1983  it  would  be  greatly  appreciated. 

I  would  like  to  add  that,  if  your  department  consents  to 
participate  in  this  study,  the  physical  therapists  who 
normally  supervise  students  in  your  department  will  be 


.. 
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informed  as  to  their  role  in  the  study;  and  they,  too,  will 
be  asked  to  sign  a  consent  form  if  they  are  willing  to 
evaluate  students  using  this  instrument.  It  should  be  noted 
that  data  gathered  during  this  study  will  not  be  used  to 
determine  students'  grades;  the  existing  evaluation  form  will 
have  to  be  completed  concurrently  for  grading  purposes. 

Your  participation  in  this  research  study  will  be  greatly 
appreciated  by  the  Department. 

Thank  you. 


Sincerely, 


DJM/dft 


David  J.  Magee,  Ph.D. 
Associate  Professor  and 
Chairman 
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Informed  Consent  Form 


Evaluation  of  Clinical  Competence  of  Physical  Therapy  Students 


Explanation  of  Field  Testing  (retained  by  clinical  affiliation 

and  rater) 

In  the  clinical  education  program  at  the  University  of 
Alberta,  fourth  year  physical  therapy  (PT)  students  are 
expected  to  achieve  the  competencies  required  for  adequate 
performance  upon  entry  into  the  physical  therapy  profession. 
The  narrative  evaluation  currently  in  use,  however,  tends  to 
be  susceptible  to  rater  biases  and  does  not  assess  if  these 
desired  competencies  have,  in  fact,  been  achieved.  A  clinical 
evaluation  instrument  should  be  developed  which  objectively 
determines  if  a  student  can  perform  entry-level  competencies 
adequately  as  judged  by  specific  standards  of  performance. 
Furthermore,  systematic  studies  have  not  been  conducted  to 
determine  the  validity  and  reliability  of  evaluations 
reported  in  the  literature  and  professional  documents.  None 
of  these  evaluations  is  well-designed  or  assess  clinical 
competence . 

An  instrument,  Evaluation  of  Clinical  Competence,  has 
been  developed  by  the  investigator  which  evaluates  clinical 
competence  as  demonstrated  by  a  fourth  year  PT  student 
according  to  the  standards  of  performance  expected  by 
completion  of  the  program.  Hence,  the  competencies  to  be 
evaluated  are  clearly  defined  and  the  four  standards  of 
performance  are  highly  specific,  as  one  means  of  ensuring 
accuracy  and  reliability  of  the  evaluation. 

The  study  in  which  you  are  being  requested  to  particip¬ 
ate  is  an  attempt  to  validate  the  Evaluation  of  Clinical 
Competence  instrument.  An  example  of  a  competency  and  its 
standards  is  the  following: 

When  implementing  the  PT  treatment  program,  the  student: 

-  is  not  observed 

-  needs  frequent  assistance/correction  to  perform 
selected  treatment  procedures  correctly  and  safely 

-  performs  selected  treatment  procedures  correctly  and 
safely  but  is  awkward  and/or  ineffective 

-  performs  selected  treatment  procedures  correctly, 
safely  and  effectively  but  is  slow 

-  performs  selected  treatment  procedures  in  a  correct, 
safe,  effective  and  proficient  manner. 

Clinical  affiliations  and  raters  agreeing  to  participate 
in  this  study  will  evaluate  fourth  year  PT  students  on  two 
different  clinical  placements  (November,  1932  and  March, 

1983)  using  two  evaluation  forms: 

1.  The  Evaluation  of  Clinical  Competence  and 

2.  The  Hiring  Rating  Form  which  indicates  a  rater's 
willingness  to  employ  the  student. 


' 
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Third  year  PT  students  will  he  evaluated  during  one  clinical 
placement  (February,  1983)  using  the  same  tv/o  evaluation 
forms . 


Each  student  in  both  third  and  fourth  year  is  evaluated 
by  two  therapists  assigned  to  supervise  the  student  at 
the  clinical  placement! s ) .  The  first  rater  is  designated  as 
the  student's  primary  resource  person  and  supervisor. 

The  second  rater  may  be  a  clinical  instructor,  team  super¬ 
visor  or  another  therapist  working  in  close  proximity  and 
in  the  same  specialty  area  who  has  not  been  assigned  another 
student  in  the  same  year  to  supervise  directly. 

Raters  are  requested  to  observe  competencies  on 
a  number  of  occasions  and,  if  feasible,  with  different 
patients  over  the  four  weeks  of  the  placement.  Raters  will 
observe  at  least  one  major  competency  area  per  day,  one 
which  is  mutually  agreed  upon  by  both  the  student  and  raters; 
then  raters  will  enter  the  student’s  subscores  on  a  progress¬ 
ive  evaluation  sheet.  The  final  score  given  to  each  competency 
will  be  calculated  from  the  multiple  entries  throughout 
the  clinical  placement. 

In  November,  1982  the  Evaluation  of  Clinical  Competence 
form  is  administered  to  fourth  year  PT  students  to  obtain 
data  by  which  to  analyse  and  revise  the  evaluation  form.  In 
February  and  March  1983  the  revised  version  of  the  Evaluation 
of  Clinical  Competence  is  administered  to  third  and  fourth 
year  students  respectively  to  determine  its  validity  and 
inter-rater  reliability. 

At  the  completion  of  the  study  students  and  raters  will 
be  asked  to  answer  a  short  questionnaire  about  their  feelings 
toward  the  evaluation  form,  its  advantages  and  disadvantages. 

You  have  the  right  to  withdraw  from  participation  at 
any  time  during  the  study.  No  records  which  would  permit 
your  identification  will  be  made  public  or  used  in  a  research 
article  without  your  written  consent.  No  information  gathered 
in  the  study  will  be  used  to  determine  the  students'  clinical 
grades;  the  existing  evaluation  form  will  be  completed 
concurrently  and  utilized  for  grading  purposes.  After 
completion  of  the  study,  students  who  request  feedback  will 
receive  by  mail  a  copy  of  their  results  on  the  revised 
version  of  the  Evaluation  of  Clinical  Competence  only. 

In  the  event  that  questions  arise  concerning  the  study, 
please  feel  free  to  contact  Joan  Loomis  (432-5968). 


•• 
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Informed  Consent  Form 

Evaluation  of  Clinical  Competence  of  Physical  Therapy  Students 

Rater  Consent  (retained  by  the  investigator) 


I  _  do  hereby 

please  print  name 

agree  to  participate  as  a  subject  in  the  study  entitled 
"An  Evaluation  of  Clinical  Competence  of  Physical  Therapy 
Students"  to  be  conducted  by  Joan  Loomis.  The  nature  of 
this  study  has  been  explained  to  me.  I  understand  that 
the  results  will  not  be  used  for  the  purpose  of  determining 
students'  clinical  grades  and  that  students  will  receive 
feedback  on  their  results  from  the  revised  evaluation  form 
only.  I  have  been  advised  that  I  may  withdraw  from 
participation  at  any  time. 


Rater's  signature 


Date 


Address 


Phone  No. 


I  was  a  witness  to  the  rater’s  signature. 


V/itness's  signature 


Date 


Please  return  to  the  investigator: 
Joan  Loomis 

Department  of  Physical  Therapy 
210  Corbett  Hall 
University  of  Alberta 
Edmonton,  Alberta 
T6G  2G4 


* 
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Informed  Gonsent  Form 

Evaluation  of  Clinical  Competence  of  Pysical  Therapy  Students 

Gonsent  of  the  clinical  affiliation  (retained  by  investigator) 


The  Physiotherapy  Department  at  the  _ 

please  print 

_  does  hereby  agree  to 

name  of  institution 

participate  in  the  study  entitled  "An  Evaluation  of  Clinical 
Competence  of  Physical  Therapy  Students"  to  be  conducted  by 
Joan  Loomis.  The  nature  of  this  study  has  been  explained. 

It  is  understood  that  the  results  will  not  be  used  for 
the  purpose  of  determining  clinical  grades  and  that  students 
will  receive  feedback  on  their  results  from  the  revised 
evaluation  form  only.  I  have  been  advised  that  I  may 
withdraw  from  participation  at  any  time. 


Director  of  the  Physiotherapy  Department  Date 


Address  Phone  No. 

I  was  a  witness  to  the  above  signature. 


Witness's  signature  Date 


Please  return  to  the  investigator: 
Joan  Loomis 

Department  of  Physical  Therapy 
210  Corbett  Kail 
University  of  Alberta 
Edmonton,  Alberta 
T6G  2G4 


■ 


. 
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Informed  Consent  Form 


Evaluation  of  Clinical  Competence  of  Physical  Therapy  Students 


Explanation  of  Field  Testing  (retained  by  the  student) 

In  the  clinical  education  program  at  the  University  of 
Alberta,  fourth  year  physical  therapy  (FT)  students  are 
expected  to  achieve  the  competencies  required  for  adequate 
performance  upon  entry  into  the  physical  therapy  profession. 
The  narrative  evaluation  currently  in  use,  however,  tends  to 
be  susceptible  to  rater  biases  and  does  not  assess  if  these 
desired  competencies  have,  in  fact,  been  achieved.  A  clinical 
evaluation  instrument  should  be  developed  which  objectively 
determines  if  a  student  can  perform  entry-level  competencies 
adequately  as  judged  by  specific  standards  of  performance. 
Furthermore,  systematic  studies  have  not  been  conducted  to 
determine  the  validity  and  reliability  of  evaluations 
reported  in  the  literature  and  professional  documents.  None 
of  these  evaluations  is  well-designed  or  assess  clinical 
competence . 

An  instrument,  Evaluation  of  Clinical  Competence,  has 
been  developed  by  the  investigator  which  evaluates  clinical 
competence  as  demonstrated  by  a  fourth  year  FT  student 
according  to  the  standards  of  performance  expected  by 
completion  of  the  program.  Hence,  the  competencies  to  be 
evaluated  are  clearly  defined  and  the  four  standards  of 
performance  are  highly  specific,  as  one  means  of  ensuring 
accuracy  and  reliability  of  the  evaluation. 

The  study  in  which  you  are  being  requested  to  part¬ 
icipate  is  an  attempt  to  validate  the  Evaluation  of  Clinical 
Competence  instrument.  An  example  of  a  competency  and  its 
standards  is  the  following: 

When  implementing  the  FT  treatment  program,  the  student: 

-  is  not  observed 

-  needs  frequent  assistance/correction  to  perform 
selected  treatment  procedures  correctly  and  safely 

-  performs  selected  treatment  procedures  correctly  and 
safely  but  is  awkward  and/or  ineffective 

-  performs  selected  treatment  procedures  correctly, 
safely  and  effectively  but  is  slow 

-  performs  selected  treatment  procedures  in  a  correct, 
safe,  effective  and  proficient  manner. 

Fourth  year  students  agreeing  to  participate  in  this 
study  will  be  evaluated  in  two  different  clinical  placements 
(November  1982,  and  February  1983  For  students  outside 
Edmonton  or  March  1933  For  students  in  Edmonton)  using  two 
evaluation  Forms: 

1.  the  Evaluation  oF  Clinical  Competence 

2.  the  Hiring  Fating  Form  which  indicates  a  rater's 


' 


' 
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willingness  to  employ  the  student. 

Third  year  students  agreeing  to  participate  in  this 
study  will  be  evaluated  in  one  clinical  placement  (February 
1983)  using  the  same  two  evaluation  forms. 

Each  student  is  evaluated  by  two  therapists  assigned  to 
supervise  the  student  at  the  clinical  placement ( s ) .  The  first 
rater  is  designated  as  the  student’s  primary  resource  person 
and  supervisor.  The  second  rater  may  be  a  clinical  instructor, 
team  supervisor  or  another  therapist  working  in  close 
proximity  and  in  the  same  specialty  area  who  has  not  been 
assigned  another  student  in  the  same  year  to  supervise 
directly.  , 

Raters  are  requested  to  observe  competencies  on  a  number 
of  occasions  and,  if  feasible,  with  different  patients  over 
the  four  weeks  of  the  placement.  Raters  will  observe  at 
least  one  major  competency  area  per  day,  one  which  is 
mutually  agreed  upon  by  both  the  student  and  raters;  then 
raters  will  enter  the  student’s  subscores  on  a  progressive 
evaluation  sheet.  The  final  score  given  to  each  competency 
will  be  calculated  from  the  multiple  entries  throughout 
the  clinical  placement. 

In  November  1982  the  Evaluation  of  Clinical  Competence 
form  is  administered  to  fourth  year  FT  students  to  obtain 
data  by  which  to  analyze  and  revise  the  evaluation  form. 

In  February  and  March,  1983  the  revised  version  of  the 
Evaluation  of  Clinical  Competence  is  administered  to  third 
and  fourth  year  FT  students  respectively  to  determine  its 
validity  and  inter-rater  reliability. 

At  the  completion  of  the  study  students  and  raters  will 
be  asked  to  answer  a  short  questionnaire  about  their  feelings 
toward  the  evaluation  form,  its  advantages  and  disadvantages. 

You  have  the  right  to  withdraw  from  participation  at 
any  time  during  the  study.  No  records  which  would  permit 
your  identification  will  be  made  public  or  used  in  a  research 
article  without  your  written  consent.  No  information  gathered 
in  the  study  will  be  used  to  determine  a  student’s  clinical 
grade;  the  existing  evaluation  form  will  be  completed 
concurrently  and  utilized  for  grading  purposes.  If  you  would 
like  feedback  after  the  study,  please  leave  a  self-addressed 
envelope  with  the  investigator.  Only  the  results  from 
the  revised  version  of  the  evaluation  will  be  mailed  to  you 
as  soon  as  possible  following  completion  of  the  study. 

In  the  event  that  questions  arise  concerning  the  study, 
please  feel  free  to  contact  Joan  Loomis  (^32-5963). 


' 


, 
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Informed  Consent  Form 

Evaluation  of  Clinical  Competence  of  Physical  Therapy  Students 

Student  Consent  (retained  by  the  investigator) 


I 


please  print  name 


do  hereby 


agree  to  participate  as  a  subject  in  the  study  entitled 
"Evaluation  of  Clinical  Competence  of  Physical  Therapy 
Students"  to  be  conducted  by  Joan  Loomis.  The  nature  of 
this  study  has  been  explained  to  me.  I  understand  that 
the  results  will  not  be  used  for  the  purposes  of  determining 
clinical  grades  and  that  I  will  receive  feedback,  if  I 
request  it,  on  my  results  from  the  revised  evaluation  form 
only.  I  have  been  advised  that  I  may  withdraw  from 
participation  at  any  time. 


Student’s  signature 


Date 


Address 


Phone  No. 


I  was  a  witness  during  the  explanation  referred  to 
above  and  to  the  student's  signature. 


Witness's  signature 


Date 


Please  return  to  the  investigator: 
Joan  Loomis 

Department  of  Fhysical  Therapy 
210  Corbett  Hall 
University  of  Alberta 
Edmonton,  Alberta 
ToG  2C-4 


- 
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October  4,  1 982 


Recently,  you  kindly  agreed  to  provide  a  clinical  placement 
for  a  fourth  year  physical  therapy  student  from  the  University 
of  Alberta  from  January  4  to  February  11,  1983.  I  am  writing  to 
ask  you  if  you  would  also  consider  participation  in  a  research 
study  that  is  being  conducted  with  fourth  year  PT  students  at 
the  University  of  Alberta  while  in  their  clinical  placements 
during  the  1982-83  academic  year. 

Parti ci pati on  by  your  department  in  this  study  requires 
two  supervising  therapists  to  evaluate  the  student's  performance 
during  the  last  two  weeks  of  the  placement  (January  31  to  February 
11,  1983)  using  an  evaluation  form  which  is  being  developed  and 
tested  by  me.  This  letter  continues  to  explain  the  study  in  more 
detail,  to  provide  informed  consent  forms  and  to  give  instructions 
for  your  inclusion  in  the  study. 

This  study  is  thes i s  research ,  with  the  approval  of  the 
Department  of  Physical  Therapy  at  the  University  of  Alberta  and 
my  thesis  committee,  on  an  evaluation  instrument  that  I  am 
developing  to  evaluate  clinical  competence  of  PT  students. 

The  development  of  a  valid  and  reliable  evaluation  of  clinical 
competence  has  future  implications  for  students,  physical 
therapy  educators  and  employers  of  new  graduates.  It  is  hoped 
that  your  physical  therapy  department  will  agree  to  participate 
in  this  much  needed  and  worthwhile  study. 

Enclosed  are  two  informed  consent  forms:  1)  an  explanation 
of  field  testing  and  2)  consent  forms  for  the  clinical  affiliaton 
and  raters  outside  Edmonton. 

The  explanation  of  field  testing  outlines  the  rationale 
of  the  study,  the  design  of  the  evaluation  instrument  and 
the  method  of  administering  this  evaluation  instrument  in  order 
to  test  its  validity  and  reliability.  Please  read  the  explanation 
carefully  and  contact  me  (432-2071  )-  if  you  have  any  questions. 

If  you  feel  that  your  department  is  able  to  assist  me  with  this 
study,  please  complete  the  consent  form  (for  clinical  affiliation 
outside  Edmonton)  and  return  to  Joan  Loomis  at  the  Department  of 
Physical  Therapy,  University  of  Alberta. 


■ 


. 
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If  your  department  consents  to  participate  in  this  study, 
two  physical  therapists  need  to  be  assigned  to  evaluate 
the  student  between  Jan.  31  and  Feb.  11,  1933.  The  first  rater 
will  be  the  therapist  assigned  to  supervise  the  student  through¬ 
out  the  placement;  the  second  rater  is  assigned  to  observe  and 
evaluate  the  student  during  the  two  week  duration  of  the  study 
only.  These  two  therapists  are,  then,  asked  by  you  to  read 
the  explanation  and  sign  the  informed  consent  form.  Two 
informed  consent  forms  (for  raters  at  clinical  affiliations 
outside  Edmonton)  are  enclosed.  Extra  consent  forms  are  also 
enclosed  in  case  of  late  changes  necessitating  a  change  of 
raters.  Please  return  the  raters'  completed  consent  forms  to 
me  along  with  your  department's  consent  form. 

At  first  glance,  it  may  seem  difficult  to  provide  a  second 
rater  to  evaluate  a  student.  However,  the  recently  completed 
pilot  project  indicated  that  the  amount  of  time  involved  over 
and  above  the  time  normally  expected  in  supervision  of  a  student 
is  approximately  15  minutes  per  day  by  the  second  rater.  In 
your  case  it  would  be  15  minutes  daily  for  two  weeks. 

It  should  be  noted  that  data  gathered  during  this  study 
will  not  be  used  to  determine  the  student's  grade;  the  existing 
U.  of  A.  evaluation  form  will  have  to  be  completed  concurrently 
for  grading  purposes. 

If  you  participate  in  this  study,  I  will  be  sending  you 
the  new  evaluation  form  and  its  instructions  in  early  January, 

1  933  . 


The  Clinical  Coordinator  at  the  U .  of  A.  has  asked  me  to 
tell  you  that  he  will  be  sending  the  evaluation  materials 
currently  required  by  U .  of  A.  in  early  December. 

Your  participation  in  this  research  study  will  be  greatly 
appreciated  by  myself  and  the  Department  of  Physical  Therapy 
at  the  University  of  Alberta. 


S  i  n  c  e  r  e  1  y  , 


Joan  Loomis 
Assistant  Professor 
Phys i cal  Therapy 
University  of  Alberta 


•  i 
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Informed  Consent  Form 


Svaluation  of  Clinical  Competence  of  Physical  Therapy  Students 


Explanation  of  Field  Testing  (retained  by  the  rater  and 

clinical  affiliation  outside 
Edmonton ) 

In  the  clinical  education  program  at  the  University  of 
Alberta,  fourth  year  physical  therapy  (PT)  students  are 
expected  to  achieve  the  competencies  required  for  adequate 
performance  upon  entry  into  the  physical  therapy  profession. 
The  narrative  evaluation  currently  in  use,  however,  tends  to 
be  susceptible  to  rater  biases  and  does  not  assess  if  these 
desired  competencies  have,  in  fact,  been  achieved.  A  clinical 
evaluation  instrument  should  be  developed  which  objectively 
determines  if  a  student  can  perform  entry-level  competencies 
adequately  as  judged  by  specific  standards  of  performance. 
Furthermore,  systematic  studies  have  not  been  conducted  to 
determine  the  validity  and  reliability  of  evaluations 
reported  in  the  literature  and  professional  documents.  None 
of  these  evaluations  is  well-designed  or  assess  clinical 
competence . 

An  instrument,  Evaluation  of  Clinical  Competence,  has 
been  developed  by  the  investigator  which  evaluates  clinical 
competence  as  demonstrated  by  a  fourth  year  PT  student 
according  to  the  standards  of  performance  expected  by 
completion  of  the  program.  Hence,  the  competencies  to  be 
evaluated  are  clearly  defined  and  the  four  standards  of 
performance  are  highly  specific,  as  one  means  of  ensuring 
accuracy  and  reliability  of  the  evaluation. 

The  study  in  which  you  are  being  requested  to  particip¬ 
ate  is  an  attempt  to  validate  the  Evaluation  of  Clinical 
Competence  instrument.  An  example  of  a  competency  and  its 
standards  is  the  following: 

When  implementing  the  PT  treatment  program,  the  student: 

-  is  not  observed 

-  needs  frequent  assistance/correction  to  perform 
selected  treatment  procedures  correctly  and  safely 

-  performs  selected  treatment  procedures  correctly  and 
safely  but  is  awkward  and/or  ineffective 

-  performs  selected  treatment  procedures  correctly, 
safely  and  effectively  but  is  slow 

-  performs  selected  treatment,  procedures  in  a  correct, 
safe,  effective  and  proficient  manner. 

Clinical  affiliations  and  raters  agreeing  to  participate 
in  this  study  will  evaluate  fourth  year  PT  students  in 
the  clinical  placement  between  January  31  and  February  11, 
1983  using  two  evaluation  forms: 

1.  the  Svaluation  of  Clinical  Competence 


■ 
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2.  the  Hiring  Rating  Form  which  indicates  a  rater’s 
willingness  to  employ  the  student 
The  revised  version  of  the  Evaluation  of  Clinical  Competence 
is  administered  to  fourth  year  students  at  this  time  to 
determine  its  validity  and  inter-rater  reliability. 

Each  student  in  both  third  and  fourth  year  is  evaluated 
by  two  therapists  assigned  to  supervise  the  student  at 
the  clinical  placement^ s ) .  The  first  rater  is  designated  as 
the  student's  primary  resource  person  and  supervisor. 

The  second  rater  may  be  a  clinical  instructor,  team  super¬ 
visor  or  another  therapist  working  in  close  proximity  and 
in  the  same  specialty  area  who  has  not  been  assigned  another 
student  in  the  same  year  to  supervise  directly. 

Raters  are  requested  to  observe  competencies  on 
a  number  of  occasions  and,  if  feasible,  with  different 
patients  over  the  four  weeks  of  the  placement.  Raters  will 
observe  at  least  one  major  competency  area  per  day,  one 
which  is  mutually  agreed  upon  by  both  the  student  and  raters; 
then  raters  will  enter  the  student's  subscores  on  a  progress¬ 
ive  evaluation  sheet.  The  final  score  given  to  each  competency 
will  be  calculated  from  the  multiple  entries  throughout 
the  clinical  placement. 

At  the  completion  of  the  study  students  and  raters  will 
be  asked  to  answer  a  short  questionnaire  about  their  feelings 
toward  the  evaluation  form,  its  advantages  and  disadvantages. 

You  have  the  right  to  withdraw  from  participation  at 
any  time  during  the  study.  No  records  which  would  permit 
your  identification  will  be  made  public  or  used  in  a  research 
article  without  your  written  consent.  No  information  gathered 
in  the  study  will  be  used  to  determine  the  students’  clinical 
grades;  the  existing  evaluation  form  will  be  completed 
concurrently  and  utilized  for  grading  purposes.  After 
completion  of  the  study,  students  who  request  feedback  will 
receive  by  mail  a  copy  of  their  results  on  the  revised 
version  of  the  Evaluation  of  Clinical  Competence  only. 

In  the  event  that  questions  arise  concerning  the  study, 
please  feel  free  to  contact  Joan  Loomis  (432-5968). 


■ 


. 

■ 
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Hiring  Rating  Form 
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Student's  Number 


Rater's  Number 


HIRING  RATING  FORM 


QUESTION:  If  you  had  a  position  to  fill,  would  you  hire  this 

student? 


ANSWER: 


1.  I  would  prefer  not  to  hire  this  student  for  our 
setting . 


2.  I  would  not  hire  this  student  until  he/she  has 
developed  greater  professional  competence. 


3.  I  would  consider  hiring  this  student  but  would 
have  some  reservations  at  this  time. 


_4 .  I  would  favorably  consider  hiring  this  student 
but  would  expect  to  provide  continued  supervision 
for  some  time. 


5.  I  would  definitely  hire  this  student  and  expect 
him/her  to  work  independently  within  a  very 
short  time. 


N.B.  Answer  the  question  with  reference  to  the  student's  competence 
in  your  setting,  not  according  to  your  department  policy  on 
hi  ring. 


. 


' 


APPENDIX  G 

Narrative  Clinical  Evaluation  Used  by  the  Department 
of  Physical  Therapy  at  the  University  of  Alberta 
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Faculty  of  Rehabi 1 i tation  Medicine 
Department  of  Physical  Therapy 


University  of 
Alberta 


SENIOR  PHYSICAL  THERAPY  CLINICAL  EVALUATION 

STUDENT'S  NAME:  _ 

INSTITUTION:  _ _ 

SUPERVISING  THERAPIST/S: 

PLACEMENT  DATES:  FROM:  _  TO: 

ABSENCES:  _  1/2  days 


PLEASE  READ  DIRECTIVE  BEFORE  USING  THIS  FORM 


GUIDELINES: 

EXCEPTIONAL 
ABOVE  AVERAGE 
ACCEPTABLE 

UNACCEPTABLE 

NOT  APPLICABLE  (N/A) 


' 
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I.  THEORY:  LEVEL  of  KNOWLEDGE  and  APPLICATION  of  THEORY  to  PRACTICE 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 
COMMENTS: 


II.  PATIENT  ASSESSMENT  -  SELECTION  of  PROCEDURES,  TECHNIQUE,  FINDINGS 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 


COMMENTS: 
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III .  TREATMENT  PLANNING 

A.  AIM  SETTING 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 

COMMENTS : 


B.  SELECTION  of  APPROPRIATE  PROCEDURES 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 

COMMENTS: 


C.  OBSERVATION  of  PATIENT  RESPONSE  and  APPROPRIATE  MODIFICATION 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 


COMMENTS: 
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IV.  TREATMENT 

A.  SAFET"  (of  patient  and  student) 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 
COMMENTS: 


B.  APPROACH  to  PATIENT 

(i)  Explanation,  Instruction,  Demonstration 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 

COMMENTS : 


(ii)  Ability  to  gain  patients  confidence  and  motivation 
EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE 

COMMENTS : 


C.  ORGANIZATION  -  of  time,  cubicle,  apparatus 


EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 

COMMENTS: 


D.  PROCEDURES 
(i)  Manual 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 

COMMENTS: 


(ii)  Electrical 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 

COMMENTS : 


UNACCEPTABLE 


UNACCEPTABLE 


UNACCEPTABLE 


t ; 
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(iii)  Mechanical  -  use  of  apparatus 
EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE  N/A 
COMMENTS : 


(iv)  Thermal 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE  UNACCEPTABLE  N/A 


COMMENTS : 
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V.  COMMUNICATION  WITH  STAFF 

A.  VERBAL 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 

COMMENTS: 


B.  WRITTEN 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 


UNACCEPTABLE 


UNACCEPTABLE 


COMMENTS: 


' 


PROFESSIONAL  ATTITUDE  -  Reliability,  Ethical  and  General  Behaviour 


EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 

COMMENTS: 


STUDENT'S  RESPONSE  TO  SUPERVISION  AND  CRITICISM 

EXCEPTIONAL  ABOVE  AVERAGE  ACCEPTABLE 


UNACCEPTABLE 


UNACCEPTABLE 


COMMENTS: 
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VIII.  EXPERIENCE  RECORD 


A.  CONDITIONS  TREATED/OBSERVED 


B.  ROUNDS/CLINICS 


C.  OTHER 
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IX.  ADDITIONAL  COMMENTS  IF  DESIRED 


X.  THIS  STUDENT  HAS  BEEN  CREDITED  WITH  SUCCESSFUL  COMPLETION  OF  THIS 

PLACEMENT. 


YES 

NO 

PLEASE  PLACE  CHECK  MARK  (^)  IN  APPROPRIATE  BOX. 


STUDENT'S  SIGNATURE:  _ 

SUPERVISING  THERAPIST/S  SIGNATURE/S: 

CLINICAL  SUPERVISOR  OR  HEAD 
of  DEPARTMENT'S  SIGNATURE: 


DATE  OF  SIGNING: 


' 
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Form  Completion  Sequence 
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Form  Completion  Sequence 


***  Importance  of  Instructions  *** 

Since  each  evaluation  form  requests  slightly  different  kinds  of 
information,  it  is  important  to  treat  them  separately.  Adherence  to 
the  form  completion  sequence  will  ensure  that  information  from  one  form 
will  NOT  unjustifiably  influence  the  information  on  another  form. 


***  Instructions  to  Raters  *** 

1.  Please  complete  the  three  evaluation  forms  between  DAY  18  and  DAY  20 
of  the  placement  in  the  sequence  given: 


FORM  COMPLETION  SEQUENCE 


1.  U.  of  A.  Anecdotal  Form 

2.  Evaluation  of  Clinical  Competence  Form  (Sections  I  and  II) 

3.  Hiring  Rating  Form 


2.  Start  with  FORM  1.  When  FORM  1  is  completed  to  your  satisfaction, 
proceed  to  FORM  2.  When  FORM  2  is  completed  sati sfactori ly , 
finish  with  FORM  3. 


3.  Complete  all  three  evaluation  forms  INDEPENDENTLY. 


4.  After  an  evaluation  form  is  completed,  DO  NOT  LOOK  AT  IT  AGAIN. 
***  NEVER  REFER  BACK  to  earlier  ratings 
***  NEVER  CHANGE  earlier  ratings 


5.  You  provide  the  student  with  feedback  on  the  results  of  the  U.  of  A. 
Anecdotal  Evaluation  Form  ONLY. 

***  DO  NOT  GIVE  THIS  FEEDBACK  TO  THE  STUDENT  UNTIL  ALL  THREE 

FORMS  HAVE  BEEN  COMPLETED. 


6.  Turn  in  all  three  evaluation  forms  to  the  Clinical  Instructor. 


Thank  you  for  your  help 


Joan  Loomis 
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Form  Completion  Sequence 


***  Importance  of  Instructions  *** 


Since  each  evaluation  form  requests  slightly  different  kinds  of 
information,  it  is  important  to  treat  them  separately.  Adherence  to 
the  form  completion  sequence  will  ensure  that  information  from  one  form 
will  NOT  unjustifiably  influence  the  information  on  another  form. 


***  Instructions  to  Raters  *** 


1.  Please  complete  the  three  evaluation  forms  between  DAY  18  and  DAY  20 
of  the  placement  in  the  sequence  given: 


FORM  COMPLETION  SEQUENCE 


1 .  Hi  ring  Rating  Form 

2.  U.  of  A.  Anecdotal  Form 

3.  Evaluation  of  Clinical  Competence  Form  (Section  I  and  II) 


2.  Start  with  FORM  1.  When  FORM  1  is  completed  to  your  satisfaction, 
proceed  to  FORM  2.  When  FORM  2  is  completed  sati sfactori ly , 
finish  with  FORM  3. 


3.  Complete  all  three  evaluation  forms  INDEPENDENTLY . 


4.  After  an  evaluation  form  is  completed,  DO  NOT  LOOK  AT  IT  AGAIN. 
***  NEVER  REFER  BACK  to  earlier  ratings 
***  NEVER  CHANGE  earlier  ratings 


5.  You  provide  the  student  with  feedback  on  the  results  of  the  U.  of  A. 
Anecdotal  Evaluation  Form  ONLY. 

***  DO  NOT  GIVE  THIS  FEEDBACK  TO  THE  STUDENT  UNTIL  ALL  THREE 
FORMS  HAVE  BEEN  COMPLETED. 


6.  Turn  in  all  three  evaluation  forms  to  the  Clinical  Instructor. 


Thank  you  for  your  help 


Joan  Loomis 
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Form  Completion  Sequence 


***  Importance  of  Instructions  *** 

Since  each  evaluation  form  requests  slightly  different  kinds  of 
information,  it  is  important  to  treat  them  separately.  Adherence  to 
the  form  completion  sequence  will  ensure  that  information  from  one  form 
will  NOT  unjustifiably  influence  the  information  on  another  form. 


***  Instructions  to  Raters  *** 

1.  Please  complete  the  three  evaluation  forms  between  DAY  18  and  DAY  20 
of  the  placement  in  the  sequence  given: 


FORM  COMPLETION  SEQUENCE 


1.  Evaluation  of  Clinical  Competence  Form  (Section  I  and  II) 

2.  Hiring  Rating  Form 

3.  U.  of  A.  Anecdotal  Form 


2.  Start  with  FORM  1.  When  FORM  1  is  completed  to  your  satisfaction, 
proceed  to  FORM  2.  When  FORM  2  is  completed  satisfactorily, 
finish  with  FORM  3. 


3.  Complete  all  three  evaluation  forms  INDEPENDENTLY . 


4.  After  an  evaluation  form  is  completed,  DO'  NOT  LOOK  AT  IT  AGAIN. 
***  NEVER  REFER  BACK  to  earlier  ratings 
***  NEVER  CHANGE  earlier  ratings 


5.  You  provide  the  student  with  feedback  on  the  results  of  the  U.  of  A. 
Anecdotal  Evaluation  Form  ONLY. 

***  DO  NOT  GIVE  THIS  FEEDBACK  TO  THE  STUDENT  UNTIL  ALL  THREE 

FORMS  HAVE  BEEN  COMPLETED. 


6.  Turn  in  all  three  evaluation  forms  to  the  Clinical  Instructor. 


Thank  you  for  your  help 


Joan  Loomis 


\ 
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Evaluation  of  Clinical  Competence  Form:  Fourth  Year  FT  Students 
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-If  you  have  observed  the  student  on  one  item  from  a  list  of  items  delineated  by  slashes,  you  determine  the  standard  oT  performance  by 
that  one  item. 

—if  you  have  observed  the  student  on  more  than  one  item  from  a  list  of  items  delineated  by  slashes  and  one  item  is  pot  formed  poorly,  you 
determine  the  standard  of  performance  in  accordance  with  the  most  poorly  performed  item. 
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Questionnaire  of  the  Reactions  of  Students  and  Preceptors 
to  the  Evaluation  of  Clinical  Competence 
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QUESTIONNAIRE 


Evaluation  of  Clinical  Competence  Form:  Fourth  Year  PT  Students 


Please  ci rcie  the  response  that  you  feel  is  the  most  appropriate. 

1.  The  number  of  subcompetencies  evaluated  in  this  form  is: 
too  many  about  right  too  few 

2.  The  content  of  the  subcompetencies  evaluated  in  this  form  is: 
relevant  mostly  relevant  irrelevant 

3.  Four  standards  of  performance  for  each  subcompetency  are: 
too  many  about  right  too  few 

4.  The  qualitative  differences  between  each  of  the  four  standards  of 
performance  are: 

too  large  about  right  too  small 

5.  The  description  of  the  standards  of  performance  by  which  to  judge 
a  student's  performance  is: 

highly  specific  specific  enough  not  soecific  enough 


6.  The  importance  of  observation  and  evaluation  in  the  following  major 
competency  areas  is: 


Key:  very  important,  important,  not  important 


patient  evaluation 

v .  i  mp 

imp 

not 

imp 

program  planning 

v .  i  mp 

imp 

not 

imp 

implementation  of  treatment 

v .  i  mp 

imp 

not 

imp 

communication  with  patient/family 

v .  imp 

imp 

not 

imp 

communication  with  other  health  personnel 

v.imp 

imp 

not 

imp 

documentation 

v .  i  mp 

imp 

not 

imp 

The  ease  of  observing/being  observed  on  subcompetencies 

under 

the  fol 

major  competency  areas  is: 

Key:  always  easy,  usually  easy,  seldom  e 

asy 

patient  evaluation 

always 

usually 

seldom 

program  planning 

always 

usually 

sel dom 

implementation  of  treatment 

always 

usually 

sel dom 

communication  with  patient/family 

always 

usual ly 

seldom 

communication  with  other  health  personnel 

always 

usua1 ly 

seldom 

documentation 

always 

usually 

seldom 

8.  The  ability  of  this  evaluation  form  to  evaluate  a  student  objectively  is: 
good  adequate  poor 


' 


' 


177 


2 

9.  The  ability  of  this  form  to  gain  a  good  picture  of  a  student's  actual 
performance  is: 

good  adequate  poor 

10.  Please  circle  the  response  which  best  describes  your  reaction  to 
the  following  aspects  of  this  evaluation  form: 

Key :  "  posi tive  ,  neutral,  negative 


observing/being  observed  daily 

positive 

neutral 

negative 

rating/being  rated  on  the  form  daily 

positive 

neutral 

negative 

rating/being  rated  ONLY  on  performance 
that  has  been  observed 

positive 

neutral 

negative 

determining  a  final  rating  by  consideration 
of  performance  throughout  the  placement 

positive 

neutral 

negative 

format  (lay-out)  of  the  form 

positive 

neutral 

negative 

specificity  of  the  standards  of  performance 

positive 

neutral 

negative 

forced-choice  design  for  choicing  a 
standard  of  performance 

positive 

neutral 

negati ve 

ease  of  completing-  the  form 

positive 

neutral 

negative 

administration  instructions  (for  research 
phase) 

positive 

neutral 

negati ve 

no  provision  for  subjective  interpretation 
of  student's  performance 

positive 

neutral 

negative 

effect  on  the  level  of  therapist-student 
interaction 

positive 

neutral 

negati ve 

effect  on  student's  level  of  anxiety 

positive 

neutral 

negative 

effect  on  student's  level  of  clinical 
performance 

positive 

neutral 

negative 

effect  on  the  therapist's  skill  in 
evaluating  students 

positive 

neutral 

negative 

others  (COMMENT)  : 
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